Circuit bending a talking alphabet toy with a MIDI upgrade!

A few years ago my son had a Leapfrog "Alphabet Pal" toy, a kind of drag-along plastic caterpillar/centipede chimera with legs that function as push buttons. Depending on the mode, pressing a leg makes it announce letter sounds, phonic sounds or colours - all in a super jolly voice - or play various nursery-rhyme type tunes.

A fun thing to do with this was to try and make it reel off absurd or mildly insulting sentences based on the letter sounds ("O, I, C... U, R, A, P",  "R, U, A, B?" oh the fun is endless..) Interestingly there is actually some censorship going on - if you enter F, U, C... it spouts out "ooh that tickles!" and refuses to go any further (really!)

I think our old one went to a charity shop years ago, but last week for no special reason I decided to try and find another one and see if could find a way to circuit bend it, and eBay did provide...

So, first thing was to take a look inside... I was immediately quite impressed with the build quality of it - everything is nicely bolted together with Philips screws - even every one of those feet is screwed on! It also feels very solid

Inside, everything is also nicely put together with no melted-plastic-welding, flimsy clips or blobs of glue, just more Philips screws.

So here is the brain PCB. That flexible cable connects all the switch pads for the legs and has 12 tracks, show that some kind of keyboard matrix is in use (since there are 20 switches to read via that connector). This is good news! I will come back to that later. For now I'll remove the plastic bar and tape the connector out of the way.

I already had a good idea of how I was going to tackle the MIDI conversion, so I started off by looking to see if there was a way I could control the pitch of the speech. When I lifted the PCB I found a through-hole resistor on the reverse (despite the fact the rest of the board is all surface mount components). Since there was no crystal oscillator visible, my thought was this resistor might be a timing component (perhaps in the factory they select a "best fit" resistor to make the voice pitch about right and solder it on as a last step)

Anyway I took a punt on this resistor being for the timing and I clipped the lead so I could experiment with a potentiometer. Sure enough, a 100k potentiometer in place of the resistor (which measured as 36k ohm) gave a good swing of pitch from shrill squeaking to bitcrusher-like digital croak. However at the faster end of the range it would cause a lock up until everything was powered off and on again. This was fixed by putting a 15k resistor in series with the 100k pot. I then made a hole in the back of the caterpillar and fitted the pot - one great thing about this toy is that its mostly hollow, so plenty of space for additional circuitry!

So, back to the keyboard matrix, and time for some probing with a bit of wire. There are 12 pads on the edge of the "brain" PCB that mate up with the flexible ribbon cable. I tried a quick wire connection between each combination of wires to see which letter the toy spoke, and quickly had a key matrix worked out

Pad 1 vs Pads 9, 10, 11, 12.... "P", "O", "N", n/a
Pad 2 vs Pads 9, 10, 11, 12.... "K", "L", "M", n/a
Pad 3 vs Pads 9, 10, 11, 12.... "S", "R", "Q", n/a
Pad 4 vs Pads 9, 10, 11, 12.... "H", "I", "J", n/a
Pad 5 vs Pads 9, 10, 11, 12.... "V", "U", "T", n/a
Pad 6 vs Pads 9, 10, 11, 12.... "E", "F", "G", n/a
Pad 7 vs Pads 9, 10, 11, 12.... "Z", "Y", "X", "W"
Pad 8 vs Pads 9, 10, 11, 12.... "A", "B", "C", "D"

so there is a matrix configuration with pads 1-8 as columns and 9-12 as rows (or vice-versa). Sticking an oscilloscope probe on pad 9 shows a positive going pulses at a little under 200Hz

So about 200 times a second the processor in the toy is sending a pulse to each of the pads 9, 10, 11, 12 in turn and reading the voltage at pads 1,2,3,4,5,6,7,8.

Each of the switches on the legs is making a connection between a row and a column, so where the "R" switch is pressed, pad 10 is connected to pad 3 via the switch and when the processor sends a voltage to pad 10 it will see that voltage appear at pad 3

Even though the switch contacts for the letters A,B,C and X,Y,Z are on the main PCB (and so don't need to be routed via the flexible cable) they are part of the same key matrix anyway and can be triggered from the same 12 pads. This will make our job easier!

To automatically trigger these switch inputs you might think we'd just put some kind of electronic switch or relay in place of the 26 buttons and control these from an Arduino etc... Well we could take this route but it would require a lot of electronics and wiring..

Much simpler is to hijack the existing matrix scanning... with just the 12 connections used on the ribbon cable we can simulate any button press from our own microcontroller - we just need to listen out for the "pulse" coming in on pads 9,10,11,12 and set the values on pins 1,2,3,4,5,6,7,8 accordingly.

For example if our "brain parasite" microcontroller wants the caterpillar brain to think "R" has been pressed it just sets pad 3 "high" whenever it sees pad 10 go high. It does needs to keep up with 200 pulses a second on each of the four input pads but even an 8-bit PIC can do that with ease.

You could use an Arduino for this, but a full size one won't fit inside the toy. A Teensy LC would do the job nicely but I am going to use my usual go-to MCU, the PIC16F1825 (on in this case it's bigger sister, the 20-pin PIC16F1829, since I need more I/O pins for all those key matrix connections)

Soldering one of these to an SOIC-20 breakout board, I have my brain parasite ready... (the header pins on the right are to connect the PIC programmer)

We are going to need to take MIDI in, so we will requires an opto isolator circuit (per MIDI standard) and a MIDI socket.

I decided to mount the opto on a separate bit of stripboard together with a 7805 voltage regulator so that I can run the whole thing from a 9V guitar pedal adaptor rather than batteries. My combined MIDI interface and power supply board is shown below

I fitted a power socket, MIDI socket and a 6.35mm jack into the caterpillar's rear end and still had space for the stripboard. The old battery leads can now be connected to the output of the 7805 reg (I put in a couple of silicon diodes in series to drop a bit of voltage as this toy usually runs at 4.5V off 3 x AA batteries).

The audio socket is simply connected to the speaker output (which makes the signal a lot hotter than I would like, but its a quick and dirty solution) and I used the break pole of the socket to cut the internal speaker when a jack is inserted.

The microcontroller board is now connected to the keyboard matrix pads using ribbon cables

The soldering on to the pads means that the flexible ribbon cable does not fit right back on, but by trimming the leading edge of the flexible ribbon back a bit and using some foam under the bracket to add extra pressure I was able to get the flexible ribbon back in place and working, so that the original switches all still function in parallel with the new microcontroller

Now everything can be reassembled. Make sure its all working before adding the base and legs again. I also removed the wheels and the rolling ball mechanism and attached some rubber feet so it will stay put on a desktop.

So obviously somewhere in that process I needed to write firmware and program the PIC. Since I want the original switches to still work in parallel with the MIDI control, I need to make sure that inactive outputs to the key matrix are not driven low by the PIC when the signal is off...

For example if we are sounding an "R", we set pad 3 high when we see pad 10 go high. However at other times we do not want to drive pad 3 low because this would prevent any other signal getting through from the switches. We don't want to have any effect at all on the state of a pad unless we are actively driving it high.

So in general, all our outputs need to switch between "high" and "no effect" - rather than high/low. We do this by setting the pins into "high impedence" mode (digital input mode). Therefore our drive method on one of our outputs (i.e. key matrix pins 1-8) is either

No output - pin is set to digital input mode
OR
Output high - pin is set to digital output mode and driven to HIGH digital output value

On my PIC this means toggling the TRIS (port direction) registers while leaving the PORT (port data) register bit at 1 at all times. On Arduino you might use separate pinMode and digitalWrite calls to do the same thing - or go to the underlying port registers for your particular board.

For reference here is my PIC Code - It is written for Sourceboost C on PIC16F1829 but should be fairly easily portable to other 8 bit PIC 'C' compilers

//

//
// INCLUDE FILES
//
#include <system.h>
#include <memory.h>

// CONFIG OPTIONS 
// - RESET INPUT DISABLED
// - WATCHDOG TIMER OFF
// - INTERNAL OSC
#pragma DATA _CONFIG1, _FOSC_INTOSC & _WDTE_OFF & _MCLRE_OFF &_CLKOUTEN_OFF
#pragma DATA _CONFIG2, _WRT_OFF & _PLLEN_ON & _STVREN_ON & _BORV_19 & _LVP_OFF
#pragma CLOCK_FREQ 32000000
typedef unsigned char byte; 


/*
VDD   VSS
RA5   RA0-DAT
RA4   RA1-CLK
RA3-VPP  RA2-SCAN0
RC5-K3  RC0-K0
RC4-K4  RC1-K1
RC3-K5  RC2-K2
RC6-K6  RB4-SCAN1
RC7-K7  RB5-RX
RB7-SCAN3  RB6-SCAN2
*/

#define P_SCAN0 porta.2
#define P_SCAN1 portb.4
#define P_SCAN2 portb.6
#define P_SCAN3 portb.7
#define P_TRISA 0b11111111
#define P_TRISB 0b11111111
#define P_TRISC 0b11111111 


#define RX_BUFFER_MASK 0x1F
volatile byte rx_buffer[32];
volatile byte rx_head = 0; 
volatile byte rx_tail = 0; 

// State flags used while receiving MIDI data
byte midi_status = 0;     // current MIDI message status (running status)
byte midi_num_params = 0;    // number of parameters needed by current MIDI message
byte midi_params[2];     // parameter values of current MIDI message
char midi_param = 0;     // number of params currently received

////////////////////////////////////////////////////////////
// INTERRUPT HANDLER 
void interrupt( void )
{

 // serial rx ISR
 if(pir1.5)
 { 
  // get the byte
  byte b = rcreg;
  
  // calculate next buffer head
  byte next_rx_head = (rx_head + 1) & RX_BUFFER_MASK;
  
  // if buffer is not full
  if(next_rx_head != rx_tail)
  {
   // store the byte
   rx_buffer[rx_head] = b;
   rx_head = next_rx_head;
  }  
 }
}

////////////////////////////////////////////////////////////
// INITIALISE SERIAL PORT FOR MIDI
void init_usart()
{
 pir1.1 = 1;  //TXIF   
 pir1.5 = 0;  //RCIF
 
 pie1.1 = 0;  //TXIE   no interrupts
 pie1.5 = 1;  //RCIE   enable
 
 baudcon.4 = 0; // SCKP  synchronous bit polarity 
 baudcon.3 = 1; // BRG16 enable 16 bit brg
 baudcon.1 = 0; // WUE  wake up enable off
 baudcon.0 = 0; // ABDEN auto baud detect
  
 txsta.6 = 0; // TX9  8 bit transmission
 txsta.5 = 0; // TXEN  transmit enable
 txsta.4 = 0; // SYNC  async mode
 txsta.3 = 0; // SEDNB break character
 txsta.2 = 0; // BRGH  high baudrate 
 txsta.0 = 0; // TX9D  bit 9

 rcsta.7 = 1; // SPEN  serial port enable
 rcsta.6 = 0; // RX9   8 bit operation
 rcsta.5 = 1; // SREN  enable receiver
 rcsta.4 = 1; // CREN  continuous receive enable
  
 spbrgh = 0;  // brg high byte
 spbrg = 63;  // brg low byte (31250)  
 
}

////////////////////////////////////////////////////////////
// GET MESSAGES FROM MIDI INPUT
byte midi_in()
{
 // loop until there is no more data or
 // we receive a full message
 for(;;)
 {
  // usart buffer overrun error?
  if(rcsta.1)
  {
   rcsta.4 = 0;
   rcsta.4 = 1;
  }
  
  // check for empty receive buffer
  if(rx_head == rx_tail)
   return 0;
  
  // read the character out of buffer
  byte ch = rx_buffer[rx_tail];
  ++rx_tail;
  rx_tail&=RX_BUFFER_MASK;

  // REALTIME MESSAGE
  if((ch & 0xf0) == 0xf0)
  {
  }    
  // STATUS BYTE
  else if(!!(ch & 0x80))
  {
   midi_param = 0;
   midi_status = ch; 
   switch(ch & 0xF0)
   {
   case 0xA0: //  Aftertouch  1  key  touch  
   case 0xC0: //  Patch change  1  instrument #   
   case 0xD0: //  Channel Pressure  1  pressure  
    midi_num_params = 1;
    break;    
   case 0x80: //  Note-off  2  key  velocity  
   case 0x90: //  Note-on  2  key  veolcity  
   case 0xB0: //  Continuous controller  2  controller #  controller value  
   case 0xE0: //  Pitch bend  2  lsb (7 bits)  msb (7 bits)  
   default:
    midi_num_params = 2;
    break;        
   }
  }    
  else 
  {
   if(midi_status)
   {
    // gathering parameters
    midi_params[midi_param++] = ch;
    if(midi_param >= midi_num_params)
    {
     // we have a complete message.. is it one we care about?
     midi_param = 0;
     switch(midi_status&0xF0)
     {
     case 0x80: // note off
     case 0x90: // note on
     case 0xE0: // pitch bend
     case 0xB0: // cc
     case 0xD0: // aftertouch
      return midi_status; 
     }
    }
   }
  }
 }
 // no message ready yet
 return 0;
}

////////////////////////////////////////////////////////////
// MAIN
void main()
{ 
 int i;
 
 // set to 32MHz clock (also requires specific CONFIG1 and CONFIG2 settings)
 osccon = 0b11110000;


 trisa = P_TRISA;
 trisb = P_TRISB;
 trisc = P_TRISC;
 latc=0xFF;

 apfcon0.7 = 0; // RX/DT function is on RB5
 
 ansela = 0;
 anselb = 0;
 anselc = 0;
  
 
 init_usart();
 
 // enable interrupts 
 intcon.7 = 1; //GIE
 intcon.6 = 1; //PEIE
  
 byte out_data0 = 0;
 byte out_data1 = 0;
 byte out_data2 = 0;
 byte out_data3 = 0;
 while(1) {
 
  // check for MIDI note message
  byte msg = midi_in();
  if(msg == 0x90 || msg == 0x80)
   out_data0 = 0;
   out_data1 = 0;
   out_data2 = 0;
   out_data3 = 0;
   if(msg == 0x90 && midi_params[1]) 
   { 
   // note on message
    switch(midi_params[0]) 
    {
    case 36: /* A */ out_data0 = 1<<7; break; //A
    case 37: /* B */ out_data1 = 1<<7; break; //B
    case 38: /* C */ out_data2 = 1<<7; break; //C
    case 39: /* D */ out_data3 = 1<<7; break; //D
    case 40: /* E */ out_data0 = 1<<3; break; //E
    case 41: /* F */ out_data1 = 1<<3; break; //F
    case 42: /* G */ out_data2 = 1<<3; break; //G
    case 43: /* H */ out_data0 = 1<<5; break; //H 

    case 44: /* I */ out_data1 = 1<<5; break; //I
    case 45: /* J */ out_data2 = 1<<5; break; //J
    case 46: /* K */ out_data0 = 1<<1; break; //K
    case 47: /* L */ out_data1 = 1<<1; break; //L
    case 48: /* M */ out_data2 = 1<<1; break; //M
    case 49: /* N */ out_data2 = 1<<0; break; //N
    case 50: /* O */ out_data1 = 1<<0; break; //O
    case 51: /* P */ out_data0 = 1<<0; break; //P

    case 52: /* Q */ out_data2 = 1<<2; break; //Q
    case 53: /* R */ out_data1 = 1<<2; break; //R
    case 54: /* S */ out_data0 = 1<<2; break; //S
    case 55: /* T */ out_data2 = 1<<4; break; //T
    case 56: /* U */ out_data1 = 1<<4; break; //U
    case 57: /* V */ out_data0 = 1<<4; break; //V
    case 58: /* W */ out_data3 = 1<<6; break; //W
    case 59: /* X */ out_data2 = 1<<6; break; //X
    
    case 60: /* Y */ out_data1 = 1<<6; break; //Y
    case 61: /* Z */ out_data0 = 1<<6; break; //Z
    
   }
   
  }
  
  // respond to key matrix scan by changing correct
  // output pin from high impedence (input mode) to
  // a HIGH signal (digital out mode, PORTC register 
  // is already all high bits)
  
  if(P_SCAN0) {   
   trisc = ~out_data0;
  }
  else if(P_SCAN1) {   
   trisc = ~out_data1;
  }
  else if(P_SCAN2) {   
   trisc = ~out_data2;
  }
  else if(P_SCAN3) {   
   trisc = ~out_data3;
  }  
  else {
   trisc = 0xFF;
  }
 }
}

//
// END
//