Nate Habermeyer, APR: 0:00

Hello, listeners. This is the Risky Planner Podcast. Thanks for tuning in. Hey, Albert.

Albert Brier: 0:10

Hey, Nate.

Nate Habermeyer, APR: 0:11

Nice to see you again.

Albert Brier: 0:13

You too, buddy.

Nate Habermeyer, APR: 0:13

Yeah, you've been busy going here and there to and fro, like you just got out of your car and came in, I did the garage, right?

Albert Brier: 0:22

Yeah, I was. We were doing our usual prep for this record, was sitting, I was sitting in my own garage in my car, just waiting for a break in the conversation, so I could come up here and sit next to my actual microphone, dodging,

Nate Habermeyer, APR: 0:36

dodging moose, and I don't know, Canada geese, or whatever is up in Calgary,

Albert Brier: 0:40

it's mostly geese. There's plenty of those out there right now. No

Nate Habermeyer, APR: 0:43

mooses,

Albert Brier: 0:44

not to my knowledge. No.

Nate Habermeyer, APR: 0:46

Well, you know, today is a special episode.

Albert Brier: 0:49

Yeah, it is.

Nate Habermeyer, APR: 0:50

You know why? I tell the

Albert Brier: 0:51

people why

Nate Habermeyer, APR: 0:52

this is the first episode that we are welcoming a guest. So, you know, welcome Roger Bradfield, our first guest.

Albert Brier: 1:01

Hey, thanks, Nate. Don't now please, please dial back the enthusiasm. Just,

Roger Bradfield: 1:08

I mean, you know me, quiet, strange,

Albert Brier: 1:11

yeah, not he's not the yippy type,

Nate Habermeyer, APR: 1:14

but he's perfect for this show. But yeah, you know what, Roger has also been busy, like you guys are busy, busy. What are you busy with?

Roger Bradfield: 1:23

I mean, for myself, travel just came back from a week's engagement with a client. Next week, go out to have another week's engagement with the same client in a different location,

Nate Habermeyer, APR: 1:33

in a maybe a sunny tropical location.

Albert Brier: 1:36

Oh, yes, that's hilarious.

Roger Bradfield: 1:39

Toronto, sunny and tropical, it's Calgary at the moment. Hey,

Nate Habermeyer, APR: 1:43

hey, hey, Toronto is beautiful right now.

Albert Brier: 1:47

Well, anything is tropical compared to Calgary right now. Honestly, like, we're, we're getting Seattle weather right now, and Seattle's getting our weather, and Toronto's getting Florida's weather, and Florida's probably getting South America's weather. I don't even know what's going on.

Nate Habermeyer, APR: 1:59

Yeah, well, I was, I was on LinkedIn, sort of prepping for this, and just looking at, I looking at some of the comments on some of the project management groups, right? Like, you can tell when AI is posting, and then AI is commenting back on the AI, and it's just a conversation full of AI, and so, yeah, as I was doing, your framing

Albert Brier: 2:23

is a little off, that's not a.. it's a exactly, yeah, exactly, yeah,

Nate Habermeyer, APR: 2:29

nothing, but it's just all noise, like it's.. it's lost to me, I couldn't.. like I'm trying to prepare for this, looking for like real comments, and I got none, I got.. well, Roger and

Albert Brier: 2:39

I were both like we're both taking steps into actually trying to care about our online presence, you know, from a professional perspective, like, you know, Roger has a bit of a following for his risk posts, I have a bit of a following for my, I guess, news analysis posts, and it's so danged unfortunate that the rise of Roger and Albert on LinkedIn also corresponds with the Rice AI. Every third post is an AI thing, you know. I got some,

Nate Habermeyer, APR: 3:08

I got some good on another topic. I got some good feedback from a friend on our show, actually, and he is a movie producer.

Albert Brier: 3:18

Okay, yeah, all right. He

Nate Habermeyer, APR: 3:20

produced the I documentary on The Tragically Hip, that's on that's streaming right now. Yeah, big test

Albert Brier: 3:29

of missed that one. Yeah, well, I mean, we're

Nate Habermeyer, APR: 3:31

American, right? Like, it's not, you know, the.. yeah, I know, but I know people, they watched it, they cried. But he was like, yeah, your production value is pretty good. We got three listeners out there.

Albert Brier: 3:40

Well, we're doing a lot with more than three listeners, honestly. So, like, one of the things, Dave, yeah,

Nate Habermeyer, APR: 3:50

Roger, maybe

Albert Brier: 3:53

listen to the show.

Roger Bradfield: 3:56

I hesitate to say yes, because I haven't listened to all. Oh my god, I have listened to some,

Albert Brier: 4:04

though. So, as long as you're a subscriber and you're downloading them all, that's all that really matters. Yeah, just go like

Nate Habermeyer, APR: 4:10

and subscribe. Let's get into it. We are working on projects, we're working on programs, we're working in PMOs, and we see shortcomings, and we start to experiment, and do you know new work with clients, and so we're going to talk about some of that,

Albert Brier: 4:28

some of that big idea stuff, right? Yeah,

Nate Habermeyer, APR: 4:30

yeah, Albert, maybe just set the stage a little bit, like, what are we, what am I, what are we talking about exactly?

Albert Brier: 4:36

Yeah, sure, so you, you, you started me off really strong there when you said, like, we're working at the PMO level, because that's, you know, we've talked before about what a PMO is and what a program is, and when it makes sense to start sharing resources amongst like groups of projects that have something in common, like that share as a common strategic objective, and that is basically what I just said, is a, you know, slightly sloppy but accurate definition. Question of what a program actually is. It's a group of related projects, and our latest, like the things that have kept Roger and me so busy lately, are mostly programs, right? Like, we're being asked by our clients to do schedule analyzes and cost estimates, estimates, estimates, and risk analyzes, and things on programs, right? And part of the reason why we wanted to have Roger on this, this particular episode is because we, we have been running up against the same challenge that a lot of consultants in our field run up against, which is that if you want to do a project level risk analysis, for example, there's lots and lots of standards for you to follow, right, they're all out, that there's a whole bunch of just within the ring fence of the AACE, there's like, what, six of, like, yeah, that are like totally different ways of doing risk estimation on your projects. The number of RPS and like formal guidance documents, things like that, in the AACE for how to do program level risk, zero. There are none.

Nate Habermeyer, APR: 5:59

And just for our listeners, it might not be put. What is an RP?

Albert Brier: 6:03

Raj, you want to take that one?

Roger Bradfield: 6:04

Yeah, so RPs are recommended practices put together by the AAC to give people like myself and Albert strenuous guidelines about how things should be done. They're developed over frequently multiple years by some, you know, real clever folk over at AAC who spend a lot of time getting peer reviews and making sure that the processes that they're putting together have been industry tested.

Nate Habermeyer, APR: 6:33

We kind of like you say projects, Albert, and like programs, like basically everything is a program, right? I mean, like a lot of things are right, yeah. You can't really, I mean, even when you say you work on projects, it's really a program. I mean, so complex, it

Albert Brier: 6:49

depends. Well, that's it, right? Because there are some projects that are pretty simple, right? There's, there's no reason to programify projects that sit below a certain size or complexity class, you know, like, what that class is. Well, it depends on the organization, but the fact is that you know, once you get above a certain size range, like, if you, if you can call your thing a mega project, there's a pretty good chance that it would be pretty easy to subdivide into sub projects, and secretly you're actually working on a program, right? So, in the, in the 500 million plus range is a pretty good bet that you're working on a program, anything below that's a toss up, but you know, still those are the kinds of 500 million seems

Nate Habermeyer, APR: 7:30

like pocket change sometimes nowadays with like projects that are getting well, exactly, yeah, like isn't like globally there's like $106 trillion worth of, you know, that are being thrown at projects over the next like five to 10 years. It's just insane. It

Albert Brier: 7:52

sounds like a big number. I haven't done my own research on that one, but, and we know also that the bigger they get, the less well they tend to do on cost and schedule performance, like mega projects are amongst the poorest performers, you know, historically on in terms of delivering on their cost and schedule commitments. So, you know, I think a big part of that reason, personally, like this is something that should probably be studied more, but you know, I may not be the exactly correct person to do that, but like I have a book for

Nate Habermeyer, APR: 8:21

you, if that's what you're interested. Well, I'd love

Albert Brier: 8:24

to read this book. Yeah, I think you already

Nate Habermeyer, APR: 8:27

knife

Albert Brier: 8:27

that one. Yes. Well, this is how big things get done. We love this book, but anyway, the point is that I think that one of the drivers for for poor performance on mega projects may very well be that they are actually programs, right, that have not been structured and estimated as programs, where you know if you assume that it's all project, then you are going to underestimate the number of interfaces between your sub projects, you're going to assume that people are marching in the same direction, singing from the same hymnal, when they're not right, and that could potentially be a real problem when it comes time to actually go execute, like if you're, if you're estimating basis was, I have a very big project, but in reality you have a program of very big projects, then you've missed something, like guaranteed, right? This

Nate Habermeyer, APR: 9:21

is not a new epiphany, like, where did this start? Like, well, actually, I have two questions. So we are doing some interesting things with several clients, right? And then I guess two is sort of where, where did you kind of have your aha moment about sort of this is the new

Albert Brier: 9:41

well, I can think of a specific point, like a specific client, where I kind of ran up against this issue of there not being a good way to roll up risk-impacted dollars. Okay, yeah, like I can think of that for sure. And Roger, while I'm talking, I bet you can find one somewhere in your, in your 5050

Roger Bradfield: 10:00

You were about to hit the same one as well.

Albert Brier: 10:02

It's very possible. So back in the day, I was the schedule and risk manager for the capital projects mission at Canadian Nuclear Laboratories, right? So I was, I wasn't so much on the execution side as I was on the management side, like I was, I was attempting to help our project schedulers and their risk professionals right to develop sensible estimates for cost and schedule on projects and to manage them right, so a lot of what I was doing was coming up with governance and attempting to apply it and reporting standards and things like that, and that meant that the roll-ups for all of the reports flowed through me, so I had to generate cost reports and schedule reports and funding reports, and those sorts of things that were aggregating everything, a lot of that was done for me, because you know it was a big interconnected system, lots of tools at their disposal, but one thing that wasn't rolling up was risk-impacted dollars, right? So they maintained a separate fund for program level management reserve, which, you know, spoiler alert is something that we recommend in the paper that we wrote for the AACE, but anyway, so they did do that, right? But how big should that fund be, right? When we're dealing with a bunch of projects that all kind of share a common goal, again, a program, right, and you're trying to roll up risk-impacted dollars, there's not that many ways to do it, so like me and another guy, a guy called Yaroslav, like we were working together to try to solve this problem, and we came up with some things that kind of were like a proto version of what Roger and I came up with, and by the way, I don't want to represent that Roger and I were the only two people working on this, like we have our lead author, actually is Rachel Fleming, who works for MBP Consulting. She's their risk service manager and is a very, very capable risk manager and estimator, and has some papers under her belt already. So she probably has a moment like this too, but when we, when we eventually like hammered out the like what the because the problem we were trying to solve was when you add probabilistic dollars to probabilistic dollars, one plus one does not equal two, right? It depends on a lot of factors, and I'm actually the not the right person to talk about that, like Roger knows more about this than I do, but like,

Nate Habermeyer, APR: 12:18

who was driving, like you talked about you guys trying to solve this problem, like, who was driving that sort of desire to solve that? Like, how was it a problem, and like, who is saying, "Hey, somebody should solve this? We're like, was this a conversation that was happening?

Albert Brier: 12:36

Well, like, a lot of complex problems that stems from a very simple question, which is like, we, we do project level funding at the P 50 level, we do program and portfolio level funding at the P 80 level. Okay, easy enough to say so. If you are the client for CNL, then you ask a very simple question at the at the monthly progress review meeting of like, oh, well, have you guys generated a P 80 value for your program yet, so there was a moment in a

Nate Habermeyer, APR: 13:04

meeting that somebody asked that question, and the room was like, we don't know.

Albert Brier: 13:10

Well, it wasn't so much like it was more like, oh, we'll take that away, we'll give you an answer next month.

Nate Habermeyer, APR: 13:14

That's what we did, more executive thing to say. That's

Albert Brier: 13:17

well, that's what you say, right? Like, when you're in one of those meetings, you, that's what you say. Yeah, let me go find

Nate Habermeyer, APR: 13:23

that number for you.

Albert Brier: 13:25

Yeah, well, and finding the number turned into inventing methods for estimating the number, because we very, very quickly realized that when you roll up those probabilistic values, it's just a, it's a, it's a bit hairy. So, Roger, can you say more on that

Roger Bradfield: 13:41

without going deep into the math? It's usually easy just to see it by example. Well, if you do the modeling exercise from scratch, this is something that anybody who's run a QRA before will find. You, you do one set of models, you do another set of models, you bring the pair of them together, and where you think you know your half million here and a half million there is going to be equal to a million, and it never does, and you know it's you always sit in a room and someone says, but why, and the unfortunately answer is usually that's just the way probability works. It's an outcome of the Monte Carlo process, and as I say, I mean,

Albert Brier: 14:30

I think there's a relatively clean answer as to why, though.

Roger Bradfield: 14:34

Just trying to remember what the proper math answer for it is. Oh, well, screw the math deep into the winds

Nate Habermeyer, APR: 14:43

kind of guide us a little bit. Roger, like, you've been, you've worked on so many projects, right? Like, you've got, you've got some.. I'm gonna put you on the spot. You got some war stories. Like, maybe tell us a story about where this kind of situation, like, went off the tracks without naming names. I think it'd just be interesting for listen. Is like to kind of get a little bit of insight into your experience and why this is so important.

Roger Bradfield: 15:06

Yeah, my realization of a requirement is is a little different from Albert's, because it was never driven by a client saying, you know, we need to understand an answer. It was driven by, you know, as a lot of things in my life by my own curiosity, so you end up doing work for a client that involves a number of projects that happen to sit in a portfolio or a program of works, and you're trying to come up with, well, how do we find the number that best represents it, understanding that one plus one does not equal two, certainly not in the risk world. What valuation you should be putting forward. And whilst going back to the earlier comments, it's very easy to find information on how to run a QRA. This is not a challenge. There is software, there are guidances, there's plenty of people who will tell you how to do it, but then you ask someone, well, how do you set this value, and it's the classic thing of ask 10 people, get 10 different answers. No one was doing it the same way. Most people weren't really doing it at all. They were doing it in a far more simplistic and additive fashion, of we're just going to gather up all the P 80s, throw them in a bucket, and call it good.

Albert Brier: 16:31

That starts to fall on its face when the individual projects that you're working on have, you know, cash values in the billions, right? Yeah, so because when you add all those pads together, because like, if the deterministic value of your project is a billion dollars and the p 50 value is one and a quarter billion dollars, there's a pretty fair bet that the p 80 value is 1.4$1.5 billion Did everybody

Nate Habermeyer, APR: 16:56

follow that? I'm just like listeners staying along with the back, it could be

Albert Brier: 17:01

like it could be a further 25% of your of your determinist of your base budget, right? So it could be like it could be anything. Also, it's driven by the act. I wish mortgages

Nate Habermeyer, APR: 17:10

worked like this. I'm just saying, Albert, going back to when you introduce Rachel, who's going to be presenting with you and Roger at AAC, like I know that you and Rachel had a conversation at last year's AAC that sort of sparked this. Did you not like, can you kind of take us back to that conversation? I think that would be really cool to talk to talk about here.

Albert Brier: 17:36

Yeah, sure. So Rachel and I met after, so she presented a paper last year about selecting the amongst the many project level methods, like selecting the best method for doing program level risk assessment, so I was definitely 100% going to go see that presentation, because it's something that I had been thinking about ever since this, this, this thing that I, you know, had to solve this with CNL because it's been sitting in the back of my mind, like that's what sparked my curiosity. Roger got there organically, good for him. He did that curiosity right here, exactly. So, I, on the other hand, more like a very ego with that kind of beard.

Nate Habermeyer, APR: 18:18

Roger, sorry, I digress. Go ahead, Albert.

Albert Brier: 18:21

Well, we'll have the hairstyles conversations. I could see it anyway. That's not the point. The point is that I really wanted to see what somebody who had, like, done the research, like done her homework, came up with for that, right? And the paper was very well researched and gave some very useful conclusions about how you would pick from amongst the different project level risk assessment methodologies to do certain things at the program level, but what it didn't do is say, and oh, by the way, you've got a program on your hands, here's how you estimate the value for it, didn't say that right, so it was basically like, oh, you want to do a risk assessment. Here's how you should do it's like, okay, but like, what happens to the numbers that you generate, right? Like, where do they go, and how do you add them together? And like, none of that stuff was addressed. So we had some side conversations, and I asked her, like, straight up, like, did you guys think about going down this road? Like, did, and she was like, yes, but we, you know, we same thing that Roger and I were talking about earlier. There's nothing in the literature about how you're actually supposed

Nate Habermeyer, APR: 19:26

to do it.

Albert Brier: 19:27

So we got to talking, and over the course of two or three days, like we ran into each other multiple times, and every time we had both been like noodling on it in our spare time and had more to add to the conversation, and we eventually decided, you know what, let's just, let's just do it. Let's just try to come up with a framework. Let's hammer one out, because we had some, like, rough ideas on the day, right. And then we, you know, on my side, I engaged Roger, because I knew he'd have some ideas about how to do the mathy bits, and indeed he did. So, between the two of us, we worked out a lot of the mathy bits. Um, you know, Rachel made sure that we weren't skiing too far off Piste, and also did the literature review for us, which was hugely beneficial, because it validated the core idea of the paper, which is that no one's actually out there telling projects and program teams how to do this.

Nate Habermeyer, APR: 20:14

So, you wrote a paper, and you're going to be presenting it, right?

Albert Brier: 20:17

We did. Yeah, so the paper we ended up writing. First thing we had to do, of course, was pull together an abstract, which means that we had to have some idea of what it was that we were actually going to write. So the three of us, Roger, Rachel, and I, we got, we got together, we pulled the abstract together. It's a combination of a literature review, a conceptual model, and the actual framework itself. It eventually got assigned number 4852 so in AACE parlance, it's going to be conference paper risk dash 4852 and the title is Contingency Management Framework for Multi Project Programs, riveting stuff. Yeah, a real page turner, Mike Drive, yeah, exactly, so, but

Nate Habermeyer, APR: 21:03

this is this paper is like a summation of a bunch of innovation that, like, your conversation with Rachel, stuff that we're doing with client, I mean, there's lots of like trial and error,

Albert Brier: 21:14

yeah, I mean, and this is in many ways just the first step, right, like you're absolutely right that you know, between between Roger, Rachel, and myself, we've had to solve these problems practically before, and we're not unique in that respect. Like, lots and lots of risk practitioners out there have tried to solve these problems to varying degrees of success. You know, many people have run program level risk assessments up until now. It's not like they were waiting for us to come up with a framework in order to do it. So, we're trying to make things easier and, like, give some kind of guidance and direction to some standardization, so that people can start testing in real-world scenarios. Like, how well does this work, and like, does this save time? Like, does it enhance the risk management paradigm across the program? Like, that's the main reason for releasing the paper about it, is throwing it over the fence to the rest of the risk community, you know, and giving them an opportunity to contribute to the to the dialog and share those experiences, because right now it's limited to the two of us, plus Rachel, you know,

Nate Habermeyer, APR: 22:08

right? Because it is a community effort. No, I get it, 100% I get, yeah, yeah,

Albert Brier: 22:13

like those, those very smart people that Roger referred to earlier, who write the AACE RPS, they're members of the AACE, like the AACE, does not have a standards writing body, like they lean on their membership for that. So, if this paper, you know, get.. and by the way, we've shared bits of it to groups of.. like, we spoke, all three of us, I believe, at the at the AACE Power Forum in April, just last week, I presented the technical elements of this paper, and by the way, Roger, thank you for feeding me the information I needed to not sound like an idiot when I spoke at the set phrase. I'm sure you'd have managed without,

Roger Bradfield: 22:50

but every bit helps.

Albert Brier: 22:52

That's very helpful. Yeah, it did help, apparently, because people really dug it, dug their teeth in, like, to this on the technical side of things, like, there's, there seems to be a hunger here for this, like, every place that, where we've, we've had these conversations amongst technical people, you know, people really want this, like, they, they want some guidance, and they want to be proven that there is a better way, right? Yeah, I

Nate Habermeyer, APR: 23:16

think the presentation that I attended, where you presented to the AAC, like members, there was so much engagement afterwards about, like, hey, did you consider this, or like, are you doing this, or what about this other thing, and like, there's lots of kind of technical debate and discussion,

Albert Brier: 23:37

there is, and if you, and if you just go tell Joe Blow off the street to go read risk 4852 contingency management framework for multi project programs. They'll probably throw a slice of pizza in your face if they don't like the pizza, maybe. Yeah, if you let you know, you'd say the lucky, you know, you mentioned Joe Blow.

Nate Habermeyer, APR: 23:54

So, like, who benefits from this, and how

Roger Bradfield: 24:01

that's a a, it's a bit of a challenging question. The, the positive, the most positive benefit I could see coming from this is it sets the conversation going down a particular path, and if enough people get interested, which, as Albert's intimated, this certainly seems to be a hunger for it. Then you know this framework, or something very similar to it, will be tried, and people will try it in real-world space, and put it in front of their clients, and hopefully we'll be able to get some results back from more than just the people that we deal with, and from there you hopefully you know you build it out and say, well, okay, it works really well, it's great, and then that gets converted over time to an RP, and then everybody in AACE and all the people who reference their RPs gets the benefit of. Having a structure that is considered a standard that then means that you know when you go along to an organization and say this is the framework we're going to put in front and we're going to use it has validity, so you know you can then the client can then take it to their management or their clients and say hey look this is the number we came up with. It's based on this, it's backed up by research, it's recommended by the AACE,

Albert Brier: 25:28

and that's a really important point, because project organization, the AACE really is a warm blanket for for large project organizations, like they, they want you to be targeting a specific standard, like frequently it's around things like schedule level of detail, estimate accuracy, and precision, even like what kinds of documents should go into your estimates as various phases, phases, like those are like the big ones in terms of like what organizations are looking for from the AACE. Risk is right there with it, though, so if you are doing a risk analysis, they want you to be able to cite chapter and verse of like why you're doing it in this particular way, so we're trying to give them the tools to basically, and you know what, from from a benefits perspective, if you want to call it, call it out in dollars and cents, like every moment you spend having to justify your homegrown solution for a program level risk analysis evaporates in a puff of smoke, if you've got an RP to back you up, right? So it should take the effort for getting your method approved and across, and also that should take that effort down, and also hopefully eliminate, you know, clever executives trying to come up with their own solutions, and saying, well, why don't you try it this way, or why don't you do it this way, or I want to see this, or I want to see that, and like you're never going to close the door on all of that, but hopefully you can at least curtail it somewhat, you know. If you have something to lean on,

Roger Bradfield: 26:47

it gives you that portability and traceability as well. You can point back at it and say this is the precise method that we followed. Why did we follow it? Because it's written down in an RP or a standard,

Albert Brier: 27:00

and you know all those RPS are scalable in some way. They all say, "Here's when you should use it and here's when you shouldn't. And we've done some of that in this paper, but that's one of the areas of future research that needs to be done. It's like, what's the break point for even using this, right? Because the most obvious other alternative is, okay, you know, if you're doing an integrated risk analysis, that means you need a schedule. If you're going to do a schedule for a program, that means you need all of the interconnections between the program or the sub projects within the program tracked, right? And if you're doing that, why not just build one giant monolithic program schedule? Well, that's the other option for doing this kind of estimation, and it falls on its face only at really, really high levels of complexity, when you've got 10s of 1000s of activities, you know, schedule activities to manage and hang risks on, it doesn't feel great to have to do that for a whole program, and then also have to do it for their sub projects, because they're all going to carry their own budgets, so like that's basically the problem we were trying to solve, so in principle, from an efficiency and benefits perspective, it also solves the problem of, well, now you have a more accurate estimate, right?

Nate Habermeyer, APR: 28:07

Right.

Albert Brier: 28:07

In principle, if it works the way that we think it does, you know, your program level estimate becomes just as accurate as your project level estimates, because it leans on all of those already mature, already battle-tested project level RPS for doing risk assessment. It's still, we're not throwing those out. We're basically saying do that and then subject it to this, and you get, you know, a program level result that is valid

Nate Habermeyer, APR: 28:35

when you guys, when you all are presenting at AACE on June 28 and 29th in Las Vegas. How do you want the audience to feel or sort of take away what's the what? Yeah, what is that

Roger Bradfield: 28:51

curiosity impact

Nate Habermeyer, APR: 28:52

that which that you want? Yeah, and the reaction.

Albert Brier: 28:55

Yeah, Roger nailed it.

Roger Bradfield: 28:57

Curiosity, we want people to go away. I'm not saying questioning the process, but certainly thinking about it, wanting to ask questions, wanting to follow up, wanting to get involved in trying to, you know, make this the thing it could be,

Nate Habermeyer, APR: 29:12

wanting to prove you wrong, maybe. Sure, you

Roger Bradfield: 29:14

know what? If they do, so be it.

Nate Habermeyer, APR: 29:17

So be because that it just improves the whole community.

Roger Bradfield: 29:20

Yeah, exactly. Whether we're right or wrong, what we can say is we've started the conversation, and by starting the conversation and trying to get people to think about this in a structured format means that sooner or later, preferably sooner, there's going to be an answer to this. Great, if it's our one, equally great if it's somebody else's that works better,

Albert Brier: 29:43

yeah, yeah, and that's that's the main thing here is like we're contributing to a pretty mature subject matter within the AAC in risk assessment, right? It's not like you know we were the first two people ever to use Safran risk before, like we're not. Not, you know, that's the reason why there was a Safran summit. So, lots of users, like just in Toronto, there were like over 50 people who showed up to hear people like myself talk about how they use Safran Risk. That's just in Toronto, right? So, you know, every every city in North America is going to have a similar community of folks who are risk literate, right, and who wants to know what the state of the art looks like, so that they can develop good estimates for their projects and programs, and they'll get, you know, and to Roger's point, if we get proved wrong because somebody came up with a better method, like awesome, like that's part of the goal here, you know,

Nate Habermeyer, APR: 30:39

so, so, if I'm a, if I'm managing a multi-project program, right, and like curiosity is what we want that response, like, what's the question that I should be asking?

Roger Bradfield: 30:52

Who are these guys, and how can we get them into our office? Yeah,

Albert Brier: 30:56

good answer. I mean, look, the, you know, I mentioned earlier that, like, we're trying to contribute to the discourse, and that's the main reason for doing this, and that is true. Okay, but you know, we also are, like, it's good to be amongst the first people to tackle a problem like this one, or at least in a structured way, because we want to put a little bit of name recognition and exposure out there, of course, we do. Like, we're interested in being seen as thought leaders in this space. So, if somebody does show up and say, 'Hey, your thing sucks, and here's a better way of doing it, then we want to contribute to that, because, like, we know enough, you know, we're not Johnny-come-lately in the risk space. Okay, we know our stuff. So, if these, if this other person comes forward and says, 'Here's a better way, in very much the same way that we didn't think of the things that they thought of. I bet there's stuff that that they didn't think of that we did, right? So, what that doesn't mean that now there's, you know, we're not doing like Newton and Leibniz, like you know, sniping at each other from across Europe about who's got the better calculus.

Nate Habermeyer, APR: 31:56

That was a great reference, but thank you. Yeah, not many people did that. Did that make me sound

Albert Brier: 32:01

very smart and well read, because that was the, that was

Nate Habermeyer, APR: 32:03

great, but my interruption probably just ruined it for everyone, but that I love that

Albert Brier: 32:08

may have done Roger had an alternative

Roger Bradfield: 32:11

with Julian banjos, but apparently that wasn't as good. Yeah, well, okay, so good,

Albert Brier: 32:18

yeah, yeah, same same level of erudite sophistication, you know, dueling band Newton Leibniz, whose name I'm probably mispronouncing, but anyway, the point is that we're not trying to set up some kind of rivalry, we want to collaborate with that person, right?

Nate Habermeyer, APR: 32:34

Yeah,

Albert Brier: 32:35

so we've published a paper, this person has lots and lots of thoughts about it, like that person becomes a co-author on the RP. It's that simple, right? Like, you know, we want to invite that conversation, but if you're thinking about, like, as an owner of a program budget, what kind of question would you ask after seeing this presentation? The question I'm hoping people ask is why,

Nate Habermeyer, APR: 32:59

right? Why do it

Albert Brier: 33:00

this way, right. Well, because on its face it looks pretty complicated. All right, we - I'm not going to get into too much of the details here, but, like, you know, I do think that we should spend a little bit of time talking about the stochastic roll up, because that's that's pretty cool, and you know it's something that Roger can riff on for probably the next three or four hours if we let him.

Nate Habermeyer, APR: 33:21

Well, can we do. You want to get into it right now? Like, maybe we can. Like, Roger, can you describe that a little bit? Well,

Roger Bradfield: 33:29

I don't know again, without going into the real detail of this one thing. I will say is that one of my roles in this was as a guinea pig, so within building the paper, Rachel and Albert obviously already had a large number of conversations about how they thought this could be resolved, how they thought they could, you know, this problem could be tackled and challenged, which meant when I came on board, one of the things they were got to

Nate Habermeyer, APR: 33:59

whack you with sticks,

Roger Bradfield: 34:01

well, no, that as well, but different subject. Basically, you know, we need to build this model, we need to come up with something, so we can try out the theory and try out the methodology, and see if we can make it work. So, you know, many days of playing in p6 and an equally large number of days of playing in Safran, we came to some conclusions. We worked some things out, but part way through that, one of the interesting things was me sitting down and having a bit of an aha moment, of you know, actually, if you set this all up correctly, it's potentially less work, the viable alternative, and the viable alternative Albert's already hinted at, which is you end up with the 100,000 activity schedule that is the combined effort of 10 different subcontractors. Plus, your own forces, plus budgets from here, there, and everywhere, trying to build a glorious Monte Carlo simulation that, yes, we have the compute power for, but chances are you haven't made a critical error in it somewhere pretty damn high. The stochastic roll up actually led us to a place where we are building sub projects in a way that we can then put them into a larger model to get a valid output, and as I said, done correctly with the right oversight and the right guidance in place, this actually becomes at least a far easier way to get to an almost identical answer.

Nate Habermeyer, APR: 35:44

I like easier, that's basic too. A lot of people do

Roger Bradfield: 35:49

efficiency, also good killing two birds

Albert Brier: 35:54

with one stone. Theoretically, I mean, people use that to mean one of that's good versus

Nate Habermeyer, APR: 35:59

two in the bush. Sure, sure, um doesn't make

Albert Brier: 36:04

killing many birds being a positive thing has always been slightly mysterious to me, but so

Nate Habermeyer, APR: 36:09

Albert was at a pretty good, like, description of, like, what you're.. I mean, I'm not even that.. I'm not technical, because, as listeners know, like, I run marketing and brand at Docanish, and to drive that sounds interesting to me. That sounds interesting, because I'm hearing sort of a data visualization that I can sort of rely on.

Albert Brier: 36:30

You talk about your own knowledge and capabilities in this space, if you don't have any, but you do, right? Like, you, you've been hanging out with us for long enough, not you get it, right? Like, you understand too long,

Nate Habermeyer, APR: 36:40

like a century of experience between the three of us, mostly I take off here exactly, but I have, yes, there's, yeah, so rubbed off,

Albert Brier: 36:53

so like you know, but but you, but you aren't a risk practitioner, and you still kind of get, oh, I see why this could be a thing, a

Nate Habermeyer, APR: 36:59

long way from frag nets,

Albert Brier: 37:01

yeah, indeed, yeah, still a favorite word. I used that word in a meeting earlier. I was thinking of you the whole time.

Nate Habermeyer, APR: 37:06

I use it in like incorrect context all the time.

Albert Brier: 37:09

No, I know I've been there. Can you pass away?

Nate Habermeyer, APR: 37:12

Frag nets, I mean basalt.

Albert Brier: 37:17

So we've been talking about a stochastic roll up, as if it's clear on its face what that means. Okay, but you know it's what I was talking about earlier, of taking all the risk-impacted estimates and adding them together, like what we've basically done is say take, you know, assuming that you've got a bunch of different sub projects that are all doing risk in their own, you know, theoretically appropriate ways, you need to find a way, or we were trying to find a way of taking those outputs and treating them as inputs into the program level risk analysis, so you don't have to throw them in the garbage, you don't have to do the whole thing over again, and you don't wind up with two separate sets of outputs, where one is for the program and one is for the projects, like you basically have one unified model has been constructed as like a set of by a set of steps like of controlled steps, right. So the the cool thing was was finding a way to do that right, and Roger ended up building a risk model that you know we were talking about having plenty of compute power for this sort of thing. It put some stressors on a pretty heavy duty computer, right?

Roger Bradfield: 38:26

Yeah,

Nate Habermeyer, APR: 38:27

really. I mean, if you're, we're talking 10,000 lines, that's a lot of data. The

Roger Bradfield: 38:32

challenge typically with doing this sort of modeling is the more random variables you throw into the program, the more complicated and the more problematic it comes, the number of activities is if none of them have variation on them, it's very simple to compute, but if every single line has estimate uncertainty, and then that's overlaid on top of specific risks, which is overlaid on top of other risks, the more complexity you build in, the more compute power you need, even if you only want some

Nate Habermeyer, APR: 39:08

computer time,

Roger Bradfield: 39:09

but even if you only have a relatively on paper small model, if you have enough risk packed into it and enough variance packed into it, your compute time goes from, oh, I can run, you know, this very simple model, and get an output in five minutes to, well, I'll be back in an hour and a half, and I hope it's done and not crashed. We've run into those a few times in interesting, trying to prove some of this stuff out.

Albert Brier: 39:36

Yeah, like there's, there's a, it's, we're asking a lot of the current state of the art, like the current level of technology, and the current, you know, computing power available to your average everyday risk analyst, right. We're pretty well topping that out, right, but we're not overtopping it, you know what I mean, like it's still scaled to fit within the boundary line. Is of your typical risk practice, and that's something that I think is a novel contribution. The thing I like about this framework the most, right, isn't the fancy maths. The fancy maths are cool, okay, but, but you get multiple things that your program should be doing anyway that will be beneficial to the program, all packed in to one framework, like it mandates, or I should say, it works best and assumes the existence of a program level, like a thick layer of management at the program level, like a steering committee, a risk practitioner who sits at the program level, who owns risk at the program level, it requires the existence of a budget that is managed at the program level, so you need thick management, right? Like, you need, you can't just have the program exist on paper, it needs to actually have a management layer attached to it, whether that's the PMO or something else, like that's down to the organization. But in order for this framework to work, you have to have all those things in place. So that's one thing that I really like about it. Another is that by virtue of building these, the stochastic roll up methods, I'm falling into the same trap as Roger was, really, like, so in order to build the stochastic roll up, you actually have to build summary schedules, right? Like, that's how it works. Now, summary schedules are, like I mentioned earlier, a little bit of a hot topic in the risk community, like, is it a good idea, is it a bad idea, can you avoid it? Like, why do it, like, all that stuff, like, it's an active source of discussion and debate, and that's fine, but whether you are doing it for risk purposes or other purposes, you're probably going to have to develop a summary schedule for your program at some point, right? So, why? Because 10,000 activities don't present very well on a monthly report, right, so it winds up getting rolled up aggressively to the point where a lot of the detail about how the work is actually flowing right from from project to project and even within projects it gets destroyed, right. It becomes hard to see at the program level, so you know I've been working on some other tooling that solves this problem in different ways, but like, if you build a roll-up schedule, for you know, to be in compliance with this proposed framework that we developed, you get us a pretty good summary schedule that also includes your cost, by the way, like, just as part of the package, right? So I envision programs using that as a cornerstone of their monthly reporting, because they've already got it right, like they had to go through all the effort of developing it, and like, how did it get developed? You may ask, well, that brings me to the third thing that I really love about it, which is that it forces you to think about connectivity between your projects, right? Like, how what do they share? Like, is one project developing land that is then utilized by another project? Are there shared labor resources? Are there shared pieces of, like, procurement or engineering, or whatever? Like, what's actually being shared amongst your program? You have to think about those things, so that you can develop the linkages between the projects and seed the risk model with data.

Nate Habermeyer, APR: 43:01

If I'm curious about, you know, all of these things, and experimenting, like, where do I start?

Albert Brier: 43:07

Weirdly, you should start in Las Vegas. Why? Because that's where the expo is. So, Roger and I, and of course, Rachel, we will all be presenting our paper, Risk 4852 Contingency Management Framework for multi project programs at this year's AACE conference and expo, and that is happening june 28 through the 30th at the MGM Grand in Las Vegas. Our paper specifically is being presented on the morning of the 30th, so don't leave early, okay? You got to come see us, you got to talk to us about it,

Roger Bradfield: 43:39

and we'll also be presenting the paper at the Saffron Expo in Las Vegas the day after the AAC conference is completed.

Nate Habermeyer, APR: 43:49

Excellent. Well, Roger, thank you for being our first guest on our podcast. Thank you for

Roger Bradfield: 43:56

having me. Hopefully the next one will be more entertaining, guest, I mean podcasts.

Albert Brier: 44:01

Roger, you're.. I can't thank you enough for all of the kind things you've said about the show. On the show, you've been really helpful. It's true, like, yeah, it's true. Frankly, listener, if you've gotten this far in the episode, thank you. You are a hero.

Nate Habermeyer, APR: 44:23

All right. Well, thanks, Albert. Thanks, Roger.

Albert Brier: 44:26

Thanks, Dave. You

Roger Bradfield: 44:26

can cut whatever you want.

Nate Habermeyer, APR: 44:28

All right. See you next time.

Albert Brier: 44:36

Hey everybody, it's Albert here. Thanks for tuning in to the Risky Planner podcast. We hope today's conversation was informative, and above all else inspires you to excellence in what you do. If you liked today's episode, don't forget to rate, subscribe, and leave a review. It helps us reach more listeners just like you. I'd also like to thank Thomson Egbo Egbo for letting us use his excellent music on our show. If you like what you hear, check him out at Igbomusik dot. Com, that's E G B O music.com Talk to you later.