Hey, it's the end of the internship! Okay, not really, since we have at least one more week. But it could be.
The old version of everything is working. Axtell and Gregor are still playing with constant q, but we do have a working FFT with peak finder, beat analysis, note generation, and statistics data. And they don't generate errors (well, unless the file you give it is error-ridden, in which case, we can't help you). I finished off the grapher for the stats data as well. It prints the average, as well as the skew, standard deviation, and spread, with kurtosis as changing colors.
Next week: code clean up!
Showing posts with label friday. Show all posts
Showing posts with label friday. Show all posts
23 July, 2010
16 July, 2010
7/16/10 Daily Journal of AT
Hey, it's a blog! There hasn't been much to say for the past two days; it mostly related to testing and tweaking numbers of BeatFinder and Threshold. They're mostly stable now, so we should be set with that. I also created a statistics finder to go with the FFT, which works, but doesn't do much except for display numbers. At the professor's request, I added a method to write the statistical data, as well as all the FFT data, to a file. (I will add more about the statistical data soon.) On a four minute song, the data size is 2.4 GB. And I got an error message:
Next week: Making it smaller! Also, graphing statistical data.
Next week: Making it smaller! Also, graphing statistical data.
02 July, 2010
7/2/10 Daily Journal of AT
Today was kind of slow. Primarily, it was spent testing different multipliers and size ranges for the Beat Finder. The multiplier is how the threshold for determining whether something is a beat or not is found (if the given data is higher than the average times the multiplier, it's a beat). The tutorial I found said that the most reliable multiplier is within a function relating to variance, but after getting a lot of screwy data, I decided to work with a base multiplier for now. Trouble is, it works differently on different files. With something with a strong base beat, like the Thunder intro, it does quite well, but on anything with a more even sound, it usually doesn't pick up the beats.
I was able to successfully implement the Bark sizing to the frequencies. Previously, when I wanted to find the beat in a given range of hertz, I used a multiplier, which worked well on low frequencies, and poorly on high ones. The Bark sizing works on a semi-logarithmic scaling, so low ranges are still accurate, but high ones can also be found. For example, here's the base beat of Thunder Intro:
The handclaps are still imperceptible ):
We also tried putting together our GUI and FFTs, and found out that we don't actually know what the height of the peaks is in, other than relative loudness. Presumably, since we're using the magnitude rather than the direct data, there is not necessarily any correlation between heights, volts, or decibels, other than relatively.
I was able to successfully implement the Bark sizing to the frequencies. Previously, when I wanted to find the beat in a given range of hertz, I used a multiplier, which worked well on low frequencies, and poorly on high ones. The Bark sizing works on a semi-logarithmic scaling, so low ranges are still accurate, but high ones can also be found. For example, here's the base beat of Thunder Intro:
The handclaps are still imperceptible ):
We also tried putting together our GUI and FFTs, and found out that we don't actually know what the height of the peaks is in, other than relative loudness. Presumably, since we're using the magnitude rather than the direct data, there is not necessarily any correlation between heights, volts, or decibels, other than relatively.
25 June, 2010
6/25/10 Daily Journal of AT
Alrighty, today was pretty good. Again, not a lot to show, but it was mostly research on finding the beats in a sound file.
Started out trying to work with the MIR files, but they made very little sense. The documentation was meager, and I couldn't actually tell what the analysis was doing. Looking at the website, I think it was the program to compare songs, but the output files wouldn't open in any given file, and looking at the txt of them, it was a bunch of numbers followed by a filename. Useful if one knows what they mean, but....
I started looking around the MIR website, and found a version of the BeatFinder written in C++. Wish I could have run it, but I don't know how to compile C++ on the terminal. Anyway, I started looking at it, and I know already, I can't translate it directly and implement it, because, rather than using an array of doubles for the data, it creates its own new object called Sounddata. All the extra information in it has already been stored in our other methods, so I see no reason to try and implement it.
Before trying to get through the BeatDetection, I saw in their comments that they had gotten their algorithms from another site, gamedev.net. I found the article they mentioned "Beat Detection Algorithms" (c) Frederic Patin 2002 , and it helped a lot. Put simply, the way the human ear finds beats is by recognizing a briefly louder sound in a song at a particular frequency. If a program can find when there are emphasized notes, it can find the beats.
It went into a lot of detail about how to compare a given sample to the average of the local sound (about one second surrounding the sample, so that if a song changes in intesity, it does not miss quiet beats and falsely return loud non-beats) to find if it is a beat or not. There are several methods of optimization, such as keeping the energy height rather than the frames of a sample, adding in a multiplier to avoid getting loud non-beats, and using an FFT to only compare the energy of given frequencies (to better find a back-beat or a cymbal hit).
Comparing what was said and what we have, I feel like we could implement this in a week (it'll probably take less than two days to write, but a while for testing various factors to become accurate). The article suggests having a logarithmic scale (and/or geometric spacing) for greater accuracy, so the constant Q should help with that. We may have to write an extra file with all the FFT data (rather than just the peaks) to use with the beat finder, but that shouldn't take up much time, programming or processing. I think it's doable in a week, doable well in two.
Started out trying to work with the MIR files, but they made very little sense. The documentation was meager, and I couldn't actually tell what the analysis was doing. Looking at the website, I think it was the program to compare songs, but the output files wouldn't open in any given file, and looking at the txt of them, it was a bunch of numbers followed by a filename. Useful if one knows what they mean, but....
I started looking around the MIR website, and found a version of the BeatFinder written in C++. Wish I could have run it, but I don't know how to compile C++ on the terminal. Anyway, I started looking at it, and I know already, I can't translate it directly and implement it, because, rather than using an array of doubles for the data, it creates its own new object called Sounddata. All the extra information in it has already been stored in our other methods, so I see no reason to try and implement it.
Before trying to get through the BeatDetection, I saw in their comments that they had gotten their algorithms from another site, gamedev.net. I found the article they mentioned "Beat Detection Algorithms" (c) Frederic Patin 2002 , and it helped a lot. Put simply, the way the human ear finds beats is by recognizing a briefly louder sound in a song at a particular frequency. If a program can find when there are emphasized notes, it can find the beats.
It went into a lot of detail about how to compare a given sample to the average of the local sound (about one second surrounding the sample, so that if a song changes in intesity, it does not miss quiet beats and falsely return loud non-beats) to find if it is a beat or not. There are several methods of optimization, such as keeping the energy height rather than the frames of a sample, adding in a multiplier to avoid getting loud non-beats, and using an FFT to only compare the energy of given frequencies (to better find a back-beat or a cymbal hit).
Comparing what was said and what we have, I feel like we could implement this in a week (it'll probably take less than two days to write, but a while for testing various factors to become accurate). The article suggests having a logarithmic scale (and/or geometric spacing) for greater accuracy, so the constant Q should help with that. We may have to write an extra file with all the FFT data (rather than just the peaks) to use with the beat finder, but that shouldn't take up much time, programming or processing. I think it's doable in a week, doable well in two.
18 June, 2010
6/18/10 Daily Journal of AT
Not a lot to say today. Sort of lost momentum from the good days we've been having. Morning was spent on NoteFinder, trying to get that to be more accurate. Used logarithmic scaling between notes, but that just made it worse. Ended up using note and next note, without anything between, and that worked best, but there's still a lot of spill.
Afternoon was spent helping with GUI (peer programming) and trying to get windows to work better. Bartlett works, but the two H's have too much noise, and increasing the peak filter hasn't helped yet. May move on to constant Q transforms, which is basically DFTs with logarithmic scaling. Of course, the papers I've looked at so far assume we know the math behind DFTs really well, so...don't know how well I'll do on that. Perhaps this weekend I shall belatedly research them, and figure out how the math works. It's one of those things that shouldn't work, but does.
Oh, and I ran Maple Leaf Rag through our GUI. Took half an hour and the data was very, very wrong. *sigh*
Afternoon was spent helping with GUI (peer programming) and trying to get windows to work better. Bartlett works, but the two H's have too much noise, and increasing the peak filter hasn't helped yet. May move on to constant Q transforms, which is basically DFTs with logarithmic scaling. Of course, the papers I've looked at so far assume we know the math behind DFTs really well, so...don't know how well I'll do on that. Perhaps this weekend I shall belatedly research them, and figure out how the math works. It's one of those things that shouldn't work, but does.
Oh, and I ran Maple Leaf Rag through our GUI. Took half an hour and the data was very, very wrong. *sigh*
11 June, 2010
6/11/10 Daily Journal of AT
Hooray! I feel like everything that went wrong yesterday has been put right. Well, some of it, anyway.
We scrapped the JASS FFT function and implemented the version from Sonogram, which works a lot better. Apparently, the underlying waves were still showing through in the JASS code, which was throwing everything off. Now, nearly everything is working as it should. The windows work, we're pretty sure correctly, although the redrawing problem still exists. The 440 FFT now works and gives the same single peak consistently, regardless of the length of the file. In addition, an older problem of the FFT not working on wave files smaller than a second has been repaired, almost certainly by changing the FFT. Now, an accurate FFT can be run on files of any length (within reason, as data on a few frames will be far from accurate, and the computer may not be able to handle files larger than 20 seconds. However, the FFT will likely only be run on files at a max of 1 second, so that's not really a problem.)
My main dilemma is with the Peak Data program that finds the peaks of a given set of FFT data. There are three options that, at one time or another today, I've implemented:
1) Use a pre-generated threshold to pick the peaks. This is rather dangerous and inaccurate, as the range of peaks generated in different data varies greatly based on the length and the occurrence of other peaks. This method will not be used.
2) Use a threshold based on the top percentage of the range of data. This would be similar to the above, but much more accurate on any given set of data. However, the issue becomes as to what percentage to take. With testing, I've determined that, in general, anything less than 10% will miss many smaller peaks, while anything larger than 20% will pick up too much noise. This method may be used.
3) Get the average of the data and delete anything smaller. Run it multiple times to leave the largest peaks. I had initially thought this would half my data each time; however, it has turned out to get rid of anywhere from half to nine tenths of the previous data, based on what was left. This seems much better for finding more accurate peaks; however, the problem of how many times to run it comes up again. From testing, it should be run between 4 and 7 times. This method may be used.
4) Some combination of the above. This is most likely what I will do, along with more accurate peak finding.
Alternatively, I should probably write a more accurate peak finder. Right now, it considers anything a peak if the data immediately before and after is less than the given value. Perhaps with checking more surrounding data and having more rigorous qualifications will allow the finder to do more of the work, and perhaps not need the cleaner at all.
The other thing I found was that the WaveSplitter is kind of slow. When the length of the resulting mini files is a second, or a largish fraction of a second (as low as .1 sec) it isn't noticeably slow. However, getting files that are 512 samples long (which is around .01 seconds) takes a lot longer. For example, on a two minute song, the program generates 9654 files, and takes about twenty minutes to generate and run an FFT on each. Adding in the analysis will probably bring the total to 30 minutes per song analyzed. Which isn't too bad, but if it can be faster, I'd like it to be.
TODO: Write function to give frequency (range) of a given peak. Write peaks to file. Sort file. Use file to print new graph of total peaks in a song to graph. Get WaveSplitter to go faster, possibly by not saving wave files (?) or by having larger sample size. Find peaks more accurately.
We scrapped the JASS FFT function and implemented the version from Sonogram, which works a lot better. Apparently, the underlying waves were still showing through in the JASS code, which was throwing everything off. Now, nearly everything is working as it should. The windows work, we're pretty sure correctly, although the redrawing problem still exists. The 440 FFT now works and gives the same single peak consistently, regardless of the length of the file. In addition, an older problem of the FFT not working on wave files smaller than a second has been repaired, almost certainly by changing the FFT. Now, an accurate FFT can be run on files of any length (within reason, as data on a few frames will be far from accurate, and the computer may not be able to handle files larger than 20 seconds. However, the FFT will likely only be run on files at a max of 1 second, so that's not really a problem.)
My main dilemma is with the Peak Data program that finds the peaks of a given set of FFT data. There are three options that, at one time or another today, I've implemented:
1) Use a pre-generated threshold to pick the peaks. This is rather dangerous and inaccurate, as the range of peaks generated in different data varies greatly based on the length and the occurrence of other peaks. This method will not be used.
2) Use a threshold based on the top percentage of the range of data. This would be similar to the above, but much more accurate on any given set of data. However, the issue becomes as to what percentage to take. With testing, I've determined that, in general, anything less than 10% will miss many smaller peaks, while anything larger than 20% will pick up too much noise. This method may be used.
3) Get the average of the data and delete anything smaller. Run it multiple times to leave the largest peaks. I had initially thought this would half my data each time; however, it has turned out to get rid of anywhere from half to nine tenths of the previous data, based on what was left. This seems much better for finding more accurate peaks; however, the problem of how many times to run it comes up again. From testing, it should be run between 4 and 7 times. This method may be used.
4) Some combination of the above. This is most likely what I will do, along with more accurate peak finding.
Alternatively, I should probably write a more accurate peak finder. Right now, it considers anything a peak if the data immediately before and after is less than the given value. Perhaps with checking more surrounding data and having more rigorous qualifications will allow the finder to do more of the work, and perhaps not need the cleaner at all.
The other thing I found was that the WaveSplitter is kind of slow. When the length of the resulting mini files is a second, or a largish fraction of a second (as low as .1 sec) it isn't noticeably slow. However, getting files that are 512 samples long (which is around .01 seconds) takes a lot longer. For example, on a two minute song, the program generates 9654 files, and takes about twenty minutes to generate and run an FFT on each. Adding in the analysis will probably bring the total to 30 minutes per song analyzed. Which isn't too bad, but if it can be faster, I'd like it to be.
TODO: Write function to give frequency (range) of a given peak. Write peaks to file. Sort file. Use file to print new graph of total peaks in a song to graph. Get WaveSplitter to go faster, possibly by not saving wave files (?) or by having larger sample size. Find peaks more accurately.
04 June, 2010
Weekly Blog Internship Update #1
As the first week of our internship, much of our time was spent learning the basics of what we were doing, and reviewing how to do it. Tuesday was our first semi-official day, where we went over the grant proposal and learned the overarching ideas behind this internship; namely, analyzing music and sound files, then creating new, similar music from the data generated.
The first analysis method we're using was the Fourier transformation, which takes wave data, finds the frequencies of the peaks, and puts it into graph form. The hope is that, once the frequencies in a song are known, they can be analyzed to figure out what instruments are playing what notes, and how often, in order to aid in further music generation. Our first official day, Wednesday, was spent learning about Fourier transformations, the related mathematical formulas, and how to perform the functions faster using FFTs.
We also started looking at different previously established programs that used FFTs to analyze music. The first was Sonogram, a "visual speech" open source program that displays sound files in colorful graph forms, which we worked with on Wednesday and Thursday. We had high hopes that, due to it's code being in Java, we could better understand how to use it and possibly implement it in our work. However, the language barrier (Sonogram was programed in German) and the number of nested unnecessary calls and transformations made the program very difficult to understand and work with, and we ended up abandoning it.
We moved on to looking at three other programs with similar functionality: JASS, Audacity, and Jigl. JASS (Java Audio Synthesis System) is used for audio manipulation, primarily as a researcher resource. Audacity's primary use is as a sound mixer and amature recording program. Jigl (Java Image and Graphic Library) is a set of functions designed for students studying digital image and audio processing. All programs are open source. Our hope was that, through these three programs, we could create and test a simple FFT program.
We came at the problem from three directions: translating the audio files (in this case, WAVE files) into bytes, processing the bytes in the FFT function, and putting the resultant data into a useful form. Our initial forray into audio-to-byte functions lead to some problems: the bytes, when put back into audio form, produced a similar but distorted audio file. Comparing Fourier graphs of the two showed that the direct copying would not give an accurate reading. Instead, we started working with the pre-built Java class AudioInputStream, which gave us grouped bytes rather than the raw data.
With analyzing the data itself, after we understood the function, implementing it was rather easy. However, we could not test it until we got the audio into byte form. Even after that, we had no way to test the numbers we got to see if they were correct or useful. For now, we compared the data to running the program through Audacity's FFT, but the data was vastly different. We're working on figuring out how to better test our FFT function, and how to begin storing and showing the data.
Our goals for next week are creating a working function to take an audio file and give a Fourier transformation, and create a method to check that our data is correct. We also hope to better understand and utalize the various error-catching functions present in the codes we have read so far, and to be able to better understand C++ code.
The first analysis method we're using was the Fourier transformation, which takes wave data, finds the frequencies of the peaks, and puts it into graph form. The hope is that, once the frequencies in a song are known, they can be analyzed to figure out what instruments are playing what notes, and how often, in order to aid in further music generation. Our first official day, Wednesday, was spent learning about Fourier transformations, the related mathematical formulas, and how to perform the functions faster using FFTs.
We also started looking at different previously established programs that used FFTs to analyze music. The first was Sonogram, a "visual speech" open source program that displays sound files in colorful graph forms, which we worked with on Wednesday and Thursday. We had high hopes that, due to it's code being in Java, we could better understand how to use it and possibly implement it in our work. However, the language barrier (Sonogram was programed in German) and the number of nested unnecessary calls and transformations made the program very difficult to understand and work with, and we ended up abandoning it.
We moved on to looking at three other programs with similar functionality: JASS, Audacity, and Jigl. JASS (Java Audio Synthesis System) is used for audio manipulation, primarily as a researcher resource. Audacity's primary use is as a sound mixer and amature recording program. Jigl (Java Image and Graphic Library) is a set of functions designed for students studying digital image and audio processing. All programs are open source. Our hope was that, through these three programs, we could create and test a simple FFT program.
We came at the problem from three directions: translating the audio files (in this case, WAVE files) into bytes, processing the bytes in the FFT function, and putting the resultant data into a useful form. Our initial forray into audio-to-byte functions lead to some problems: the bytes, when put back into audio form, produced a similar but distorted audio file. Comparing Fourier graphs of the two showed that the direct copying would not give an accurate reading. Instead, we started working with the pre-built Java class AudioInputStream, which gave us grouped bytes rather than the raw data.
With analyzing the data itself, after we understood the function, implementing it was rather easy. However, we could not test it until we got the audio into byte form. Even after that, we had no way to test the numbers we got to see if they were correct or useful. For now, we compared the data to running the program through Audacity's FFT, but the data was vastly different. We're working on figuring out how to better test our FFT function, and how to begin storing and showing the data.
Our goals for next week are creating a working function to take an audio file and give a Fourier transformation, and create a method to check that our data is correct. We also hope to better understand and utalize the various error-catching functions present in the codes we have read so far, and to be able to better understand C++ code.
Subscribe to:
Posts (Atom)