10.6: Pixel Neighbors – Processing Tutorial


Ok, so here’s another important aspect about image processing that we left out in the previous video. (clears throat) So what did we do? We had, an image of a frog we processed every single pixel and we copied that pixel to our processing display
window and maybe we manipulated it while copying it, we made it brighter
we changed it’s color, we did all sorts of things. This is, I would say, pixel point processing. What we are doing is there’s a one to one relationship, for this pixel change this pixel for the next pixel, change this pixel for the next pixel, change the next pixel. However, most sophisticated image processing algorithms that you might find in not to keep using photoshop as an example photoshop or a ** image processing. actually process goups of pixels for example, let’s say this is my frog pixel my frog image and this here is my processing window, and let’s say this is the pixel I want to set here in my processing window. What if inorder to set this pixel
I actually used a whole bunch of neighbouring pixels around that pixel, so that pixel is over here I don’t know what am drawing here exactly
umm but I actually took all of the pixels in five by five matrix around it and
this is the roo.. the color of this pixel is a function of all of these pixels
combined. Ahh aah, a classic image processing algorithm that you might have used before is a blur. What if you take a pic.. an image and blur it making it fuzzier, less sharp, less in focus. Well actually if you average all of the pixels of an area to set a particular pixel, that’s a blur algorithm. It is making each pixel more like it’s neighbours almost ahh bluring it, averaging it, smoothing it between pixels. So this is a actually an algorithm and sharpening it is the opposite taking each pixel and making it more different than it’s neighbours. Umm another example
of an algorithm is finding the edges in an image.
So if I have a silhouette of a person (chuckles) this looks nothing like a person and I don’t know.. some weird ghost alien thing and I wanna find the edges of this person well let’s say this person was wearing a black shirt, much like the.. my black shirt here how would I find the edges? Well, if this is if you’re looking at an image of me the pixels that are very different the white pixels and these black pixels a pixel that’s very different from it’s neighbours is an edge right? (finger taps his microphone twice) this pixel over here is very similar to all it’s neighbours. This pixel right here is quite different than it’s neighbours it must be on an edge. So, finding the edges of an image has to do with looking for pixels that are very different from their neighbours. Let’s actually see if we can, implement this really really quickly in our frog example. (taps to switch cameras) So if I come over here, to the frog and if I look at this froooooog and what where we doing here? We were ahh kinda doing this threshold thing, that’s where we last were. Well, let me say, let’s aah take all this out and let’s take all this out. Here is a particular pixel location, what if I want to look at this pixel relative to the pixel to it’s right or to it’s left, or above or below. Well, what will that pixel be this is location1 another location might be (x+1) times y times width and if you look at this what is the difference here? Now am saying, let me look at pixel x + y xy and let me look at pixel x+1, y right? Here ahh if am over here and then am in some image if this is pixel x – oh you can’t see that. If this is pixel x, y then this right here right next to it is pixel x+1, y so ofcourse a more sophisticated algorithm migh look at x+1 and x-1 and y+1 and y-1 x+2 and y+2 aloot of pixels in the neigboring area but just to start really simply let’s look at this pixel and the one next to it. (taps to switch camera) So if I’m looking at this pixel and the one next to it, I can look at both of the brightness of both of those pixels. b1 equals frog.pixels[loc1]; b2 equals frog.pixels[loc2]; Aaah those cameras are always shutting off am gonna fix that in the middle of this video look I turned black for a second and am back and am just going to fix this one you’ll have to bear with me, there’s nothing I can do about this right now. ohhh come back (taps twice) Ok, am back I’l edit all that out later ok so now uhh I could get a difference which is maybe am just gonna say the absolute value of (b1-b2) and let’s set the pixel of the window equal to that difference noow, oops loc1 so loc1 is kind of my core original location so I wanna set that individual pixel but I wanna look at it as a function the value of that pixel as a function of two pixels that are next to each other let’s run this and by the way we got this sad, very difficult, really traumatic array out of bounds exception this error is something that will happen to you as soon as you start doing any image processing work. What happened? I tried to look at, investigate, think about access a picel that doesn’t exist, because at some point I was at a pixel that was all the way at the edge of my image and I tried to look at pixel one more to the edge which doesn’t exist. So there’s alot of strategies around this problem for right now am just going to do something which is kind of simple which is say let me just look at all the pixels all the way up to one before the
edge and that will kind of like deal with this problem but you can think of other things you could have pixels wrap around there’s lots of other ways you could approach this problem but now we can see you know this isn’t the perfect demonstration but I’ve kinda really got something here the the frog by the way was had a lot of similar colors so am getting ummm umm actually am not exactly getting what I expected I guess the frog actually didn’t have lots of similar colors it’s very different cuz we’re getting we’re getting a very bright color everywhere there. I guess there’s a big difference of all the pixels inside the frog and the background is very noisy so there’s a couple of things we’re noticing here; one – this isn’t such a great algorithm am only looking at the pixels to the left coming to the
right and to the left if I looked at the pixels above and below and diagonally I might be able to uhh determine alot more about the image the other thing is – this particular image has alot of noise in it so I’m am there’s so much noise that am not really getting accurate edges maybe blurring the image first and then looking at it looking to the edges might
help, but in the end am actually visually able to see something here but this wasn’t the exact demonstration I was hoping for but uhh to.. look at this, maybe try to oohhh I know what the problem is huuuuuh (sighs deeply) am gonna have to re-record this video but what did I just do here?
I did the thing that I did originally which is just I looked at the arbitrary
difference between the giant integer value you’re probably all just watching this
like yelling at the screen like you’re doing it wrong, what I wanted to do was look at the brightness of each one of this pixels just forgot like I wanna compare the brightness of each one of these pixels. This is gonna give us something much much more umm oh yeah and it’s very hard to see here but actually this really worked. I’m really getting the edges I’m really getting the edges cause the edges of the frog that’s where the pixels are most different and infact if I want to be a bit more thoughtful about this I might just come up with some threshold like if the pixel values are at all different then, draw black otherwise draw white and now if I run this, we can see aah I’m really starting to see the contours of this frog. I think this could be a better demonstration. So here we’re seeing just a really basic algorithm for looking at the comparison between two pixels and finding the edges now again this was a very simplistic approach most of the time what you wanna do wooah haha all that work and this camera is not even on umm most of the time what you wanna do is look at a whole selection of pixels and in fact this is something that you might not be aware of this is right there bring up photoshop for a second if I go to photoshop and I’ve got my frog loaded in here and I go to filter ahh ahh ah other custom what I dont.. what your actually seeing here is an and this is hard for you to see is this are the weights of a pixel If I wanna make a pixel on the center much more important than the ones in the area, that’s a sharpening. I want the pixel to be more different than it’s neighbours I actually ahh repr.. ahh oops I created very quickly a blur and you can see here this this would be a blur, what if I take all the pixels in a huge neighbour space, add them all together divide them by 25 as an average and you can see here that this umm frog is now blurred am gonna uncheck the preview let me kind of zoom in here umm you can see ahh over here look over here, that frog is blurred that frog is not blurred so this particular algorithm is blurring that image by adding all the pixels together and infact umm you know am not gonna go through the code for this in detail but we can look at this right here so if I look at this particular example this this particular example you can see here if I zoom in to it that the pixels inside the rectangle are being blurred and the pixels outside are not you can see kinda all the arm above me as I move the rectangle over am kinda blurring out that pixel those pixels if I go back to the code, and I put in these weights this is actually sharpening, again I want the pixel on the center to substract the pixels around it to be more different than the pixels around it and if I run this this is a bit more obvious, we can see that this is me sharpening that image now there’s alot of noise is happening here am very zoomed in but you can see how this is changing the quality of the pixels so comparing the pixels and its neighbours this idea of looking at a pixel and looking at the ones that are right of it or above or below is an important powerful technique in image processing and we can just you know this video can go on for hours and hours and look at every image processing algorithm that photoshop uses that anyone uses and we can see this technique in alot of them. I guess what am saying here is see what you can do with this idea. What if you umm what if you ahh umm try to recreate this edge detection algorithm but actually use more than just the two pixels that are to the right and left what if you use the pixels above or down what if you actually were able to interactly select whether you’re looking at the comparison of vertical pixels or horizontal pixels there’s lots of possibilities you might explore and try here and actually something that I didn’t do which I think will be really interesting to see which I would like to maybe add to another video at some point and I.. think of it as what happens when we’re processing the pixels this pixel goes here this pixel goes here what if I start to move the pixel around? what if I take, what if every time I look at a pixel I actually offset put it in some like random location near where it originally was this will actually give me almost llike a watery effect because the pixels start to like look like they’re dispersing or not. This this is an idea that I’d love to demonstrate so I think this video was already quite loong so am not gonna do it here umm but you can think of this as an excercise and I’ll try to add one in at some point too that does this. So here are the things that are missing umm am gonna upload these so if you’re watching these in the first pass this is where it’s gonna stop but what I wanna get to in the next set of videos, number one I wanna look at moving the pixels around which will also be relevant to the idea of what happens if we just sort of like forget about this way of doing things entirely it’s like this one to one pixel idea what if I just have a system where am drawing you know patterns and shapes to the screen maybe I draw grid of rectangles or I draw random circles everywhere or I draw swirls swirly spirals all over the screen what if just use an image essentially a database of color so what if I could like look up colors doing form a drawing algorithm create a painting for an image that’s one thing I’m gonna look at and once we’ve done that, the next piece that I wanna look at is what happens when this images aren’t static what if I’m getting images from a camera that are changing live real time images or images from a movie file. So those are the I guess there’s like three more three/four more videos that need to come to kinda finish off all of these concepts. Ok so send me your feedback, umm I’ll be happy to add more things, delete things, redo things or just go away not come back. (laughs) Ok so have good.. enjoy your morning, evening, night. I don’t know when you’re watching this
and you’re just like eating some cereal at home that’s… something
you might be doing and enjoy that cereal (laughs) talk to you
later. Subtitles by the Amara.org community

29 Replies to “10.6: Pixel Neighbors – Processing Tutorial”

  1. great tutorial,shiffman.thx so much for your video lessons.
    I'm a iOS developer from china and dont know if there's some publish house would like to translate your new book,that will be much helpful for beginners in china to access digital visional art and great features.
    There's no comments blow this video,so I leave it here,cause I watched from the 1st episode and really appreciate your lessons.

  2. @3:21 do you have something with this already done? Im trying to find a body "human" in a body detection similar to face detection but for detecting humans rather then animals in motion sensing/capture and build a class from this function just like @6:33 and so that i can share with others and use with other programs i might build in the future.

  3. thank you very,very much for all of your videos ! You're doing a great job allowing people far away (like me) to access really high class education!

  4. //using what you taught in this video, I made a cool 90's-esque "3D" render of an image.
    //if you copy this example, you will have to set some image in your sketch folder to //"corgi.jpg" or reset all the "corgi" variables with your own.

    PImage corgi;

    void setup() {
    size(680, 453,P2D);
    corgi = loadImage("corgi.jpg");
    background(0);
    }

    void draw() {
    loadPixels();
    corgi.loadPixels();
    for (int x = 0; x< width-20; x++) {
    for (int y = 0; y < height; y++) {
    int loc1 = x + y*width;
    int loc2 = (x+20) +y*width;
    float r = red(corgi.pixels[loc2]);
    float g = green(corgi.pixels[loc1]);
    float b = blue(corgi.pixels[loc1]);

    pixels[loc1] = color(r+20, g, b+20);
    }
    updatePixels();
    }
    }

  5. hello sir
    is it possible for using another weights value in weighted mean filter, instead of 1 in case we have more than one image

  6. Hey Daniel, loving these videos. Honestly as someone in a completely unrelated profession and trying to learn programming for fun (even went to an intro boot camp) your videos are the best I've found.

    I have a rather trivial question (I think). I am making a color picker program that spits out the RGB value of the pixel under you cursor. I have it fully functioning but I am pre-loading one image as my test image. I am hoping to be able to use a "select image" button so the users can just load any image from their machine. How would I go about doing this?

    Thanks again.

  7. Nice video you've made here. I actually got the basic of Image Processing in this video. Thanks a lot! 😀

  8. +The Coding Train
    In C, is there a header file, like #include <name_of_header_file.h>, that lets the user call the functions that you were using in your video?

  9. Hi Daniel, I got a question. When you are applying the blur effects, you say you are adding pixels and then then dividing them, evaluating such a mean value: are you talking about pixels' rgb values?

  10. sir which software you are using to code.. can u please give some description how to install and use it.. please… do v get that software in windows…??

  11. I always comment you are amazing.. still dont have anything different to say abt this effort. I have one request plz do a video about partical connection annimation that is viral these days. You can find it with partical.js with red background. Here is the link https://youtu.be/qK3cgD09Qf0. Plz do this in processingjs.

  12. Sir can you find a shortest path between two points in a image using A* algorithm in it?please sir give a video on this sir

  13. This is the most amazed I've ever been looking at a frog's crotch xD Thanks for your tutorials, man. I know i'm a bit late, but you're awesome!!!

  14. Coding train when I try doing the one where I and(b1-b2) I see no Change in my picture it keeps on saying array out of bound exception even though I -the width by 1

  15. Do you mind if I drop the source code in the comments? Its here for now, if its innapropriate please remove it.
    For all those who which to try it out please download p3.

    https://processing.org/download/

    Choose the version apropriate to your machine. Its very simple to run, unpack the download, and you are good. Don't forget to add the image file to your project folder, by clicking Sketch –> Add file.

    There are some slight variations in variable names, but otherwise everything works fine.

    I shall add the frog convolution code later.

    PImage frog;

    void setup() {

    size(640, 360);

    frog = loadImage("frog.jpg");

    //background(255);

    //image(frog,0,0);

    };

    void draw(){

    int w = width;

    int h= height;

    loadPixels();

    frog.loadPixels();

    // if using l3 x has to start at one otherwise it will produce a out of bounds error

    for (int x = 1; x < w-1; x++){

    for (int y = 0; y < h; y++){

    // variables can be changed to better names

    int l1 = x + y * w;

    int l2 = (x+1)+y*w;

    int l3 = (x-1)+y*w;

    float b1 = brightness(frog.pixels[l1]);

    float b2 = brightness(frog.pixels[l2]);

    float b3 = brightness(frog.pixels[l3]);

    float diff = abs(b1-b2);

    if(diff>20){

    pixels[l1] = color(0);

    pixels[l2] = color(0);

    pixels[l3] = color(0);

    }else{

    pixels[l1]= color(255);

    pixels[l2]= color(255);

    pixels[l3]= color(255);

    }

    float d = dist(x,y,mouseX,mouseY);

    float m = map(d,0,200,2,0);

    float r = red(frog.pixels[l1]);

    float g = green(frog.pixels[l1]);

    float b = blue(frog.pixels[l1]);

    pixels[l1]=color(diff);

    }

    }

    updatePixels();

    //noLoop();

    }

  16. Very good explanantion. I loved it.
    I have a question though: I am working to detect when a video stream have problems and start to pixelate. I am using some c code to get frame by frame and modify it. I am abble to invert the image or apply a sepia effect to every frame.

    I am using ffmpeg in pipes (one pipe capture the stream, frame by frame) and the other receives the frame, modify it and send back to the stream.

    The problem is that I am searching for any help on detect when things go bad and the stream gets pixelate, so I can restart the script for that channel.
    Can you give me some clue on what I can do or where I can find such answer?

    One observation: I am using a Linux machine and c language.

    Thank you very much

  17. These tutorials are amazing. You've literally taught us how to code our own Photoshop :DDD I took what you've taught, messed around, and was able to print colors values so I could take pictures of my friends and change their hair color using conditionals! hahah amazing.

  18. Thanks for this amazing video! 
    Question: why did I get unexpected token for float diff? I'm pretty sure I typed in exactly the same code…

  19. I added the North, South, East, and West pixels for the difference algorithm, and I'm not sure that it looks right. Very washed out I'll include the code as well. It will require the processing video library to run.

Leave a Reply

Your email address will not be published. Required fields are marked *