Podcast Sample provided by The Data Crunch Podcast

Thanks for Checking Out My Demo!

I am a smart AI driven podcast editor. Check out the sample podcast and corresponding transcription.

Below is a sample of some of the feedback I give to my podcasters. I give you contextual help so that you can quickly get your podcast published.

Umbegone Says…

  • If you see it means I removed an um (vocal filler) from this podcast.
  • Click on a word to edit the demo transcript.
  • Click the export buttons to download the demo transcript or audio.

Umbegone Demo

Removed: -151
This is a demo... No changes will be saved.

Speaker 1
Thanks yeah well that's awesome, well thanks for joining the show and I just want to start out briefly just, you know, learning a little bit about your background and   how you got into what you're doing now.


Speaker 2

     Sure, yes I   originally started started my career in   telecommunications   bring voice and data services   into   institutions in the learning institutions and then what I realized is is that you know connecting   universities and for profit schools you know connecting them online really created a huge opportunity for learning   and really crossing barriers to learn and really meeting learners on their   terms with online learning courses that kind of brought me through this journey with using technology to really make better decisions   in learning   in knowledge   and how we do that effectively and that has started up a sixteen year career focused   on that. Using data, using   e-tools to really to make it better learning environment for everybody   and make us more effective in the way that we gather information and retain information and that brought me   into   several areas- one is in the learning sciences is how do you how do you deliver learning content more effectively but also in the assessment side as well we're happy you measure   what folks are learning effectively and painlessly   and that's brought me on this   journey into the assessment industry and really making sure that every exam that's delivered in classrooms or whether it's   a licensure exam is as fast and as fair as possible and using data to be able to do that so really mitigating the   risk of human bias when it comes to measuring   humans abilities   which is   a troublesome area right? 

Speaker 1
Yeah, I know. You say effective and painless.  I know most people hate taking tests. So tell me, tell me how you approach that.

Speaker 2 

Yeah well I think there's a lot of ways I mean I think one of the most important ways that you make the test faster, right,? You know in 19   79   I was the chairman of assessment systems and helped create a technology called computerized adaptive testing. What that uses is algorithms to gauge what you know and what you don't know and then basically tailoring the content that you see, the next item you see gets more progressively difficult or progressively easier depending on your ability. And what that does is that it reduces test time by about fifty percent. We see that with the ASFAB exam that's given to our servicemen and women to make their testing experience faster and fairer and really we're starting to see that really across the world   with measurements. It's really making those exams tailored to what to the person's ability   which is really, really important. 

You know what you don't want to do is, you don't want to give one test that doesn't change to everyone- that's really, really inefficient. You know, if I'm going through the test and I know it, I know the content really well, I just fly right through it, but I still have to take that two hour exam, it doesn't change based on my ability, right? That's a problem. We probably already know over the firs,t you know, fifteen minutes if you have   knowledge of that content area   if you don't, the exam has to adapt. The   measurement tool has to adapt so it makes it fair.   That technology has been around for forty years but we still struggle   on widespread adoption of the technology for a number of reasons but that's one of the first ways you do it.   The second way you do it is right after folks take the exam you start running algorithms over it, you start running what's called psychometric   algorithms over the data set to understand   how the items are performing, how that test item performs with different audiences. What you don't want to happen on that examination is an item that's on there that you know that no one's getting right   or you don't want to, or conversely you don't an item that everybody's getting wrong. You want to make sure that's a fair, a fair item   and the way that you do that is through data modeling,   through the data sciences which is the first real you know real truly better data sciences psychometrics which is the measurement of   human intelligence.

Speaker 1

Got it. And let's let's dive into that a little bit because it's a super interesting topic and I didn't know about it until we talked so tell me a little bit about psychometrics. I mean, it's old and it's been around for a long time and   so tell me how that developed and and maybe recent advances or maybe were still using the same stuff that's been around because yeah, and it's been around 

Speaker 2
It's been around a very very long time, it's been around over a hundred years.   Charles Spearman started originally   was the father of factor analysis and what he started to do was trying to gauge the understanding of intelligence and what does that mean and really applying data to it to make sure that they were understanding those   results which started with you know with that whole concept and then how do you come up with the examinations and what does it mean when someone takes the examination? How do we know that that's accurate? How do we know this actually gauging in measuring   intelligence? So that started over a hundred years ago and then has worked through   evidence-based research for that entire time and so they were schools where folks go   they're called psychometricians and these are people with PhDs in tests and measurement and what they do is that they work through a   number of psychometric models and research models to determine if people understand that content or better and faster ways of assessing content and they use all kinds of     models. I mean it started with dimensional reduction   and then algorithms to start predicting performance of the data   again, one of the oldest data sciences   and it started with gauging human knowledge and human performance.

Speaker 1
That's interesting and   so this actually allows you, what you're talking bout is a dynamic test right? So they, as they're taking questions, you're then predicting what the next question should be said to just gauge their intelligence on a certain topic.

Speaker 2
That's right, I mean in using   evidence based research to do that, right? Not arbitrary decisions. So data models that are behind the scenes say okay you got this item wrong, you got this item correct, and then really it changes based on individual performance, which is really, really important right? I mean, you know, one size doesn't fit all in measurements and that's a really, really important part of it. If you don't have that it makes it very, very hard to gauge effectively,   intelligence   or mastery of content.

Speaker 1 

Sure so let's dive in a little bit. I'm curious how you even approach training a model like that.   Obviously there's a lot of domain expertise here   I'm assuming, but how do you approach training a model that can dynamically gauge intelligence and and give appropriate questions?

Speaker 2

 Well actually, there's a couple things. So you have to start with the modern test theory. In modern test there he is we're using   training set to carry that the calibrate the sample that's been around for very very long time the idea of   of looking at the individual item and then gauging   what that means and then the the   the algorithm that goes around it so it's really it's an algorithmic   approach to   to testing looking at that one item how did you perform and then tailoring another item in in understanding the depth of the difficulty of that it really starts by field testing. So every single time someone takes a test there's going to be in on scored item in that test you're going to gather data based on people's performance on that item right because it's on score and you're going to see how folks are performing is the item difficult is it not difficult and that's how you start training the models and then you start creating your examination from that and start building it based off of that but really starts. With with modeling it over live data and understanding how human beings perform on it it's really an important element of it and so you know that's that used it's always best practices and psychometrics is really applying it to to live to live examinations in on scored after we call field testing.

Speaker 1
That's awesome and   and you mentioned earlier I mean the science behind this is fascinating and   but you mentioned earlier there are some other roadblocks in terms of applying this and getting adoption for it. Can we go into that?

Speaker 2 

Yeah I think you know we, this industry has been around a very, very long time it's about a seventeen million dollar addressable market and it's   filled with very, very large corporations that   have vested interests with keeping examinations long. I'll give you an example. If you're charging   per hour to attest sponsored it test Bonsor would be somebody that owns the test I'll give an example   so we're talking about the   certified   financial planning exam. That's the folks that   certified financial planning board of the folks at on the examination but they're not the one to administer the exam. Folks that administer the exam will be a company like   Pearson education   Pearson DO or pro metric these very very large   assessment providers and they're gonna charge so sponsors based on the length of the examination so the longer the exam is the more revenue you make per examination. The smaller the exam less less money you can make- that's the model in the industry that's how it works and so there's a vested interest and not making it shorter right   definite vested interest in it and then you have on top of that you know this this idea where they say oh it's too difficult to do it's too complex to do that isn't the case I mean there is software out there that helps really move this process forward in a really excelerated manner I mean you can it doesn't take somebody with the spread sheet crunching numbers we have software that can do that we have software that can do it better and faster and more precise in human beings could ever do it and that's really the big difference there   but there's definite definitely a   a problem with with moving these forward again it's it's happened I mean this this technology was created in 1979, a year after I was born I mean we're still not seeing widespread adoption across   across the globe I mean we're seeing it it's funny we're seeing it outside the U. S. even more rapid adoption number seeing it in the U. S. which is a pretty pretty scary thing   you know we put on a   an annual conference we have at this   in about a month in Minneapolis. And it's computerized computer adaptive testaments to readers come together from all over the world and a large large portion of our audience is from overseas and bringing this technology to emerging markets emerging countries   it's very very interesting.

Speaker 1

Got it and to tell me   you've seen obviously some like what are the results of people that adopt us this testing approach and   what like, what kind of benefits to the see fromit?

Speaker 2 

Oh geez I mean just a   myriad of benefits. So first of all you can get a more precise instruments right? So you can make an instrument in assessment that   is more precise engaging competency and also you can reduce time quite significantly we just did that we converted   in national science exam overseas to a computerized adaptive tests and we saw that the test length was reduced by 74% which is significant.

Speaker 1

Don't know if can you complain about that.

Speaker 2

That's right, I mean you know it's testing without tears you know all of a sudden you don't have to spend eight days testing I mean it breaks my heart you know even in my personal life when I see my son who is eleven years old spending three days in testing when I know there's absolutely no reason for it. But you can do that in an hour you don't have to do it in four hours of testing you just have to construct the test more intelligently and you have to eat leverage you have to leverage technology you have to leverage   computerized testing and computerized adaptive testing.
And that's a really really important part of what we're doing there has to be a push here is a nation to move our testing online I mean ninety percent of the assessments that are delivered to test that are delivered in the state of Texas are still delivered this paper and pencil example. We're still printing paper and we're destroying paper and we're destroying forest for absolutely no reason yeah. I mean we're we're better than that right?

Speaker 1

And   and how how's that going forward? I mean obviously, there's some economic interest at play maybe some political interests at play. How do you overcome some of these   more human challenges to actually allow the data science to work?

Speaker 2
I had to use a   sort of   you know really an unpleasant analogy is- how do you eat an elephant one bite at a time and so what you do is you take you take smaller projects you bite them off and then you publish white papers. You start really putting out in the community and start eliminating those barriers   one of that one of the challenges of folks are always going to run into as human beings we intrinsically don't like change   it's something if they think something is working they have to be able to see a painless way to do it   that's one of our goals is to create applications and tools that make it easy for folks to convert make it easy for folks to convert from a linear examination which is just a plain old task,    to more adaptive measurement tool and that is   you know that you pick up small little parts you know this the the one example that I gave you with the national science competition is one example where we took a linear examination and we made adaptive in our hopes are that once you do projects like that then you start biting off more of the larger projects you know the the national examinations right you know the the test the kids taking kindergarten all the way through twelfth grade they're engaging there   mastery of the content that they're learning during those formative years you start applying those all the way through but you start with smaller projects and you show the efficiency and you show the effectiveness and then get you help educate the community and help educate test takers you know when when you go take a test which you will take and I will take we all take tests you have to ask yourself every time you take it isn't there a better way of doing this why am I going to spend six hours taking a test, haven't you folks thought of a better way.
This seems like a lot of time out of my life and that's the one resource we can't make up this time sure so starts with the individual and it starts with us with us at asking why I told my son to use that example you know ask your school why. Why do you have to take three days out of your classroom instruction to take a test when it literally can be done in an hour. But this is just the it starts with informing the public and the public actually saying you know what listen this is crazy I'm not going to take the test it's going to take six hours out of my life.

Speaker 1
That's really interesting bit by bit   that's like that so and you've talked about   you mentioned last time we talked   about a lot of of the algorithms   what you're doing is working in our right and and doing   a lot of that that work there open source so so what's going on   you know the skills been around forever has there been a recent   developments or recent breakthroughs maybe that   that you guys are finding?

Speaker 2
Well geez I mean you know the our packages have have just you know there's been just an amazing amount of innovation so that really anybody can start learning are publishing in our my   yeah my data scientist has that been sort of a it's been against R. for a very long time because you can feel that the applications are really had grown up to really support it but now I mean we're seeing folks be able to publish our packages in the connecting date TI's and all kinds of really really amazing things are happening in that community and really you know really taking taking innovation and putting it out there for everybody to see and then helping you know help looking at those models in the building I'm from Ardmore scalable technologies but starting in our I mean whether it's it's just that that is an amazing amazing thing whether it's our studio or any other applications that are out there it's making it considerably easier for folks to a learn the language and be start really really working at some of the   and some of the algorithms and some of the models it's amazing yeah.

Speaker 1
Yeah and talking about that   what do you look for when you're building out your data science team are you looking for PhD's are usually you know like what kind of skills   work in the psychometrics field?

Speaker 2
Well in our in our field primarily it's   it's PhDs and psychometrics but more precisely it's   it's quantitative psychometrics so there's you know the industry's psychometrics there's application based psychiatrist and that are you know more they'll go in and the work with an organization at sort of helping them understand the models and then there's a smaller subset of those books which are quantitative psychiatrist ins and they're the ones that actually build the models   so we're looking for quantitative folks   it but again with the very very specific focus on   on test and measurement because again this is been around for a very long time what we're seeing you know data science as an emerging as a merging of course of study in academia psychometrics a quantitative psychometrics has been around for a very very long time university of Kansas university of Minnesota these these institutions attached wanted data psychometric schools for very very long time   and so we have we have an advantage over lots of other areas that we actually have   have schools putting out candidates no it's not enough   because it's they're they're small cohorts of students maybe twelve a graduating class thirteen graduating class   but it's it but at least there's been you know a a   a focus their   and in some of the other data sciences we're just not seeing it I mean now we're trying to you know now now that the team is trying to catch up and we're seeing this pattern right that's existed for at least the last fifteen twenty years record name is spending their time trying to catch up with the emerging workforce I in that it becomes a very very large gap to try to try to   cover we're talking about complex things like machine learning   internet and you know intelligent algorithms sure   but we have an advantage that way.

 Speaker 1

That's really interesting   that's interesting   so we're coming up on time here   maybe if we could just respect your time and let you get back to your day.   Well where do you want to be in five to ten years? Like, you know, where can this idea of intelligent assessment take us? 

Speaker 2
Well I I'd read like to be in five years is that I'd like to be in a place where my son isn't spending eight hours on testing I'd like to be in a place where   you know testing becomes faster and fairer and we don't just eliminate testing what we're seeing is that with the common core and some of the push back with testing because the tester so long   and he become   they become really painful for for students and for educators that people just want to throw them out and and that's the way to do it what you do is you just make them better and make them faster so you know my hope is that you know we will live in a world where we're we're assessing more intelligently in faster and we're also you know providing content remediation to people faster and I think that we're we're I think this can happen we seen record breaking investment in educational technology over the last five years and it's continuing to grow we're seeing you know it's a world wide field   words you know the globally folks are interested in measuring ability in measuring content mastery so I think to you know I think it's a world together will be able to to really to to cross this castle   and so I've you know I've see up you know five years from now when my son is   sophomore in high school hopefully that eight hour exam is down to an hour for down to a half an hour maybe even down to fifteen minutes.

Speaker 1
I mean let's not get too crazy but if you can do it effectively, sure. That's awesome yeah absolutely all right well thank you so much for joining us I hope you are successful because   that's   that would that would be great that   I don't have to take these insanely long test anymore thank you so much Chris wrapping everything and I appreciate it have a great day.