How can one use Dragon NaturallySpeaking to code more efficiently?

The question was unfortunately closed but feel free to leave comments or answer at https://www.quora.com/How-can-we-use-Dragon-naturally-Speaking-to-code-more-efficiently. You can also vote to reopen this question if you have enough rep. Thanks!

I'm looking for ways to augment mouse and keyboard control with another form of input.

We use Dragon to trigger macro expansion. For instance, to create a voice command to write a loop template in Python, we can use the following Advanced Scripting command:

Sub Main
    Clipboard("for i in range():" & vbCrLf)
    SendKeys "^v{TAB}"
End Sub

To return the variable that is in the clipboard in Python:

Sub Main
    originalClipboard = Clipboard
    Clipboard("return " & originalClipboard)
    SendKeys "^v"
    Wait(1)
    Clipboard(originalClipboard)
End Sub

enter image description here

To add a title and a legends to a figure in Matlab:

Sub Main
    Clipboard("title('')" & vbCrLf & "xlabel('')" & vbCrLf & "ylabel('')")
    SendKeys "^v"
    SendKeys "{Enter}"
End Sub

And so on. I'm looking for any information (tips/ideas/libraries/etc) to use Dragon NaturallySpeaking to code more efficiently.

Franck Dernoncourt

Posted 2012-06-24T22:17:22.403

Reputation: 427

Programming is an activity that relays a lot in visual feedback: syntax highlighting, compiler errors in red, breakpoints, formatting. So if you need to be watching the screen Why do you try to do it without a keyboard? In fact the bottleneck should be in your mind unless you are really productive. – borjab – 2016-01-19T12:05:38.237

Although this wasn't exactly your question, there is some research into speech dictation of code, for disabled people, but it's not mature. See e.g. http://research.microsoft.com/en-us/um/people/abegel/qual/node16.html. See also related question http://programmers.stackexchange.com/questions/47933/programming-by-dictation.

– D Coetzee – 2012-07-16T01:27:56.997

Thanks a lot for the links! – Franck Dernoncourt – 2012-07-16T01:47:11.840

Answers

Alexandria Carstens, who is a speech recognition (Dragon) trainer & consultant in Vancouver, gave me a couple of advice:

Any or all of the following techniques:

Start with a blank vocabulary and add required verbiage to it (definitely take advantage of written form and spoken form).

If also require a fairly basic vocabulary as well, start with a small vocabulary, and add required verbiage to it.

Make use of text commands for repetitious text.

Make use of computer control commands (i.e. step-by-step, Advanced VB scripting) for speeding up processes such as compiling etc.

(if you are a Dragon NaturallySpeaking user, you will probably be interested in her website which gives useful advice)

Also, here is an interesting video/demo on programming by voice (using Dragon NaturallySpeaking + DragonFly): http://pyvideo.org/video/1735/using-python-to-code-by-voice

Summary

I dictate my code using a voice recognition system with Python embedded in it. In a fast paced live demo, I will code a small system and deploy it without touching the keyboard. I hope to convince you that voice recognition is no longer a crutch for the disabled or limited to plain prose.

Two years ago I developed a case of Emacs Pinkie (RSI) so severe my hands went numb and I could no longer type or work. Desperate, I tried voice recognition. At first programming with it was painfully slow but, as I couldn’t type, I persevered. After several months of vocab tweaking and duct-tape coding in Python and Emacs Lisp, I had a system that enabled me to code faster and more efficiently by voice than I ever had by hand.

The source code for voice commands isn't online yet, it should be released in June or July on https://github.com/tavisrudd?tab=repositories.

Another nice resource: Can I write SQL using speech recognition?

Franck Dernoncourt

Posted 2012-06-24T22:17:22.403

Reputation: 427

Is this up to date? Source still "not online"? – samthebrand – 2014-04-11T06:21:13.143

@SamtheBrand Did my previous comments get deleted? – Franck Dernoncourt – 2014-06-03T20:03:44.763

were there previous comments here? (I vaguely do remember this being the case.) – samthebrand – 2014-06-04T01:27:06.807

@SamtheBrand Actually I see I replied in the comments following Alex' answer, that's why I got confused :) – Franck Dernoncourt – 2014-06-04T02:50:39.447

I think the power and potential of voice activation tools lies in using spoken word to trigger actions and events, but not to replace typing.

For example, if you can configure a voice recognition package to trigger commands like "open", "save", "build", "run", etc., then you can save yourself time hopping through menus or typing hotkeys.

However, I wouldn't recommend putting any effort whatsoever into using voice recognition to actually generate your source code keyword-by-keyword. Unless you are a very fast speaker or a very slow typist, you probably won't be able to increase your speed, accuracy, or overall efficiency much by this method. Imagine having to say "dot", "open-bracket", "semi-colon", and so on every time you need the special markup characters that are so much more prevalent in programming than in prose. Without a highly specialized approach (one that will take much more effort than tweaking a package like NaturallySpeaking), I doubt you will increase efficiency.

One of the most promising areas of increasing efficiency as a programmer is in avoiding context shifts. For example, doing as much as possible without removing your hands from the keyboard (using hotkeys instead of mousing) helps programmers stay focused and effective. A voice activation system might be effective if it reduces time using the mouse, exiting your active program, but trying to replace typing with speaking does not sound likely to make coding any easier.

asfallows

Posted 2012-06-24T22:17:22.403

Reputation: 1 756

Thanks! Reducing mouse usage is indeed a key to increase productivity: I am using SmartNAV to move the mouse cursor (http://superuser.com/a/435266/116475) and I am currently looking for a device to perform mouse clicking without using hand/foot/voice (http://www.quora.com/Computer-Hardware/What-is-the-best-device-to-perform-mouse-clicking-without-using-hand-or-foot).

– Franck Dernoncourt – 2012-06-25T20:03:31.737

1For mouse clicking, an option you might want to consider is AutoHotkey - it's a system that allows you to write simple scripts including hotkeys that replace some keystrokes with others. You should be able to find a hotkey that you can map to send left-click, right-click, etc. – asfallows – 2012-06-25T20:15:35.070

1In fact I'm already using hotkeys to emulate mouse clicks, as well as foot switch, voice recognition and dwell clicking. But I would like to do it without using hand/foot/voice. I am therefore looking into brain computer interactions and eye tracking. I know there is also mouth-operated mouse, I have never tried this but it sounds pretty intrusive. I am not aware of any alternatives but I would love to hear of other devices if they exist. Maybe I should open a question on the matter here. – Franck Dernoncourt – 2012-06-25T21:07:40.627

More efficiently in what sense?

IMO voice input for programming is substantially less efficient than typing, particularly in an decent editor with good macro and template facilities. I think you'd be better served figuring out the domain(s) you normally work in, figuring out what code gen tools you already have access to or can create, and nail down every possible keyboard and template efficiency you can find.

Programming vocabulary is specialized and focuses on a different set of terminals than "normal" speech does. Context- and language-sensitive editors have access to the AST of whatever language, environment, and framework you're working on--Dragon and similar voice input software does not.

This is something I've played with every couple of years for over a decade now, and it rarely takes more than an hour or two before I give up in disgust. The abstractions we deal with as developers simply don't map to the English vocabulary. Even assuming voice macros I have yet to see a system, or combination of systems, that doesn't make me want to punch my monitor in the face.

Update

After reading this article I still mostly believe the same thing.

Where I might be starting to relent is in architectural and pseudo-code, where you could build up blocks of "this is what I will do once I start coding", although I don't have a working system I can use to test this.

Dave Newton

Posted 2012-06-24T22:17:22.403

Reputation: 1 522

Thanks for your feedback Dave! What I meant by efficiently is having the program done as quickly as possible. – Franck Dernoncourt – 2012-06-24T22:31:36.130

1@FranckDernoncourt Voice just doesn't cut it then, AFAIC, although some languages may be more amenable to voice than others. IMO not worth it at all, at least not yet. – Dave Newton – 2012-06-24T23:18:06.993

Maybe Dragon software is much better than Window's Voice recognition software.. but I can hardly get Windows Voice to do things normally! I imagine it would be hell to try and code with. – Gaʀʀʏ – 2012-06-26T01:33:37.080

1@le_garry Voice recognition is pretty good these days. I used it for this reply, in fact. But for curly-brace languages, no thanks. still, I could see it being useful for higher level language is perhaps. just not yet. – Dave Newton – 2012-06-26T01:40:03.673

Voice recognition works extremely well. Demonstration in French: http://youtu.be/FODUScE6-Ek (hopefully I'll be able to do the same soon in English). Here is an excerpt of an e-mail I received today: "amazingly after just the intro session with dns12 (on dilbert) he could read a paper abstract with heavy scientific jargon and get 99% accuracy (two words missed)!". Google voice recognition has also become very good.

– Franck Dernoncourt – 2012-10-11T01:56:33.240

which has nothing to do with programming via voice. – Dave Newton – 2012-10-11T03:38:39.380

Well it can be useful to have a good voice recognition engine for programming via voice. Unfortunately, many people such as le_garry aren't aware of the accuracy of cutting edge voice recognition engines (otherwise I guess more people would have looked into programming via voice). Hence my previous comment. – Franck Dernoncourt – 2012-10-11T17:34:36.093

I wrote a system using a similar approach to Tavis', and have been using it for a few weeks. Basically it runs Dragon with NatLink and Dragonfly in a VM and executes the actions on my Linux system. It's definitely still duct tape coding, but I'm quite happy with coding in it. https://github.com/calmofthestorm/aenea. Many of the grammars should be of use with just Dragonfly on Windows with only minor changes.

Am I faster dictating than typing code? No, but I am faster with a combination of voice and keyboard, and at any rate I spend enough time thinking anyway that a small slowdown at code entry due to voice doesn't have much of an effect on overall productivity (and I'm sure with more work I can improve)

Tristan Hayfield also recently posted about a similar project on the Dragonfly group: https://github.com/TristenHayfield/damselfly

You may also want to take a look at VoiceCode; I played with it a bit and while it wasn't really what I was looking for it's a very interesting project.

There's definitely a lot of promising stuff floating around, but I feel like at present you need to be prepared to do some substantial customization and hackery of your own in order to get a system working you'll be happy with.

I really hope Tavis releases his code soon. That video was astounding and I'm still not sure how he does some of those things (like that list comprehension).

Alex

Posted 2012-06-24T22:17:22.403

Reputation: 141

Hey great answer, but which of Tavis instances impresses you most. Why? And can you link to a particular Tavis project that you're talking about? – samthebrand – 2014-04-11T06:19:07.530

@SamtheBrand The source code for voice commands isn't online yet, it should have been released in June or July on https://github.com/tavisrudd?tab=repositories but from what I can see it's yet another conference's vanished promise... Btw do you know to which SE I can migrate this question? It got recently closed on this SE, and my last question that got closed here was deleted afterward without any warning, with no respect at all for user created content. Otherwise I'll take everything to Quora.

– Franck Dernoncourt – 2014-04-11T15:23:04.420

Hi @FranckDernoncourt. I believe this is the right site, though I suppose Programmers could host it as well. I think your best bet is to flag it for reopen. Give that a try? – samthebrand – 2014-04-11T15:26:15.437

@SamtheBrand OK I'll give a try, we'll see what happens... in the meantime I wish there was some application or web service to alert me whenever one of my StackExchange question is put on hold / gets on closed.

– Franck Dernoncourt – 2014-04-11T15:35:43.877

2@SamtheBrand 3 days after nobody else than me cast a re-open vote. I think that barely anyone is interested in voice recognition here, and I think something else but that would be banned as a comment. – Franck Dernoncourt – 2014-04-14T15:48:09.803

I've used it extensively for a code type task, in the end I had many macros that would save me 10 or 20 mouse clicks and moves! It would take me a bit less time and I found it extremely convenient.

It was for coding, something called reaktor DSP environment. I know the program pretty well. I do believe that it would be EXTREMELY efficient at coding.

What I think you need to do though, is not use any normal text. Just make a specialised command mode that doesn't interpret any normal phrases, and copy into it about 2000 words that you use in programming; "foreach, var, Vector3()" etc.

You'll need to make some commands like logical if if(){}

loop ALPHA to loop a++

After that you will be coding 1 language A LOT faster I think, unless you use predictive code typing which is very fast anyway. But you will have to train it as a command mode only, ignorning normal prose text and teach it a mass of code words. I will do this myself soon because my hands need a rest from coding for a week or 2. It will be a safe solution for me.

Remember, Dragon naturally speaking PRO is the only serious solution for making a very specialised command mode. You will be able to add commands like reference search to open a browser and read refs about a highlighted text, etc. Premium version is limited.

ants

Posted 2012-06-24T22:17:22.403

Reputation: 11

Thanks for your answer! Just 3 questions: 1) Can we create a custom mode in Dragon NaturallySpeaking? 2) How do you deal with variables (how do you call them) ? 3) Dragon NaturallySpeaking sometimes add automatically spaces: how do you avoid this (e.g. if you say "variable name open bracket" there might be a space between the variable name and the bracket, which you probably want to avoid)? – Franck Dernoncourt – 2012-10-11T01:42:13.743