I went to Japan last September and here's what I found useful.
I do know a little Japanese from a course I've done but it was a few years ago so I remembered little to none. Enough to basically get very confused any time I heard anything in Japanese.
For example as soon as I walked into my first cafe and they wanted to seat us I managed to just about ask for a table for two and then the waitress asked something I completely didn't understand. I just stood looking confused and we were taken to a table
I worked out later it was "Smoking or No Smoking" Something that it had never even occurred to me to learn.
Not speaking Japanese isn't much of a problem though as most people speak at least a little English (and often get quite excited about a chance to use it for real ^^). Also most places have either a menu with picture of what you're ordering or an English version of the menu. I was staying in Tokyo so I also got a guide book to the city, with a few other useful phrases in the back (that were rarely used, but it was nice to have them written down just in case). I would definitely advise getting a guide book, it was most useful thing I took.
In the end I settled on a few phrases I did know which served me very well.
Here is my Japanese very useful phrases list. I'm not giving exact translations, just enough to cover their use in the situation outlined. For help on how to pronounce Japanese worlds have a look here (http://www.abcb.com/misc/pron_00.htm)
sore wa = that. Very useful for ordering off menus etc. Work out what you want (or just guess) and then point out the items to order with sore.
Use: sore wa *point*
to = and. Use it instead of wa for ordering lots of things.
Use: sore to *point* sore wa*point*
Other more general useful words:
ohayo gozaimasu = good morning
konnichiwa = good day
konbanwa = good evening
sayonara = good bye
sumimasen = excuse me. To get someone's attention.
gomenasai = sorry
domo arigato gozaimasu = thank you
hai = yes
iie = no
The MOST useful thing for me was knowing Katakana. It is one of the Japanese alphabets and is only used to writing foreign words imported into Japanese. As these words are almost all English if you can read Katakana you can understand almost anything written in it (and there is a lot of katakana around). If you want to try learning it try here (http://www.kanjisite.com/html/start/kan ... ana01.html).
The only problem is because Japanese pronunciation is a lot more restricted that ours and their written system doesn't work in the same way ours does (theirs is a lot simpler, and thus much easier to learn) you have to sound out the word, context will often help when you try to translate things (eg if its on a menu you know it's going to be some kind of food).
A few words to give you the general idea:
ｶﾗ (Kora) = Cola (Coca-Cola)
ｵﾚﾝｼﾞ ｼﾞｭｽ (orinji jusu) = Orange Juice
ｱｲｽｸﾘﾑ (aisukurimu) = Ice Cream