Introduction

Resilience is now part of the lexicon of the humanitarian and development agenda (e.g. FAO-IFAD-WFP 2015). Dozens of programs and projects are implemented every year (mainly in low-income countries) which aim at strengthening the resilience of targeted populations. Understandably, governments, donors and practitioners in the development community are keen to document the effectiveness of those programs and in particular to determine whether the various activities implemented under those projects are (or not) truly effective at improving the resilience of the beneficiaries. Measuring resilience and assessing the impact of those resilience projects is however a complex and difficult exercise.

One first type of challenges relates to what could be seen as “conventional” impact evaluation problems. Among these are the difficulties associated with evaluation of on-the-ground development interventions (Ravallion 2008; Khandker et al. 2010). Because development or humanitarian projects are generally not just isolated interventions that are delivered in a perfectly controlled environment, but instead made of series of activities implemented in a ‘real-life’ context among many other external factors that also affect beneficiaries, establishing cause and effect is generally difficult (Gertler et al. 2016). Second is the well-known issue of selection bias. Because the interventions implemented as part of those projects are usually purposively targeting specific groups of vulnerable individuals or households, the condition of random selection required for true experimental designs is usually not fulfilled (De Janvry et al. 2011; Gertler et al. 2016). In those conditions, one would generally have to adopt a quasi-experimental design (Ravallion 2009; Khandker et al. 2010).

Combined to those conventional evaluation issues is the fact that the impact pathways of these projects are structured around the concept of resilience. Implementing robust impact evaluation of resilience interventions is not yet well established (Béné et al. 2017). Part of the challenge is that, although some progress has been made on what resilience is and how to conceptualise it—see e.g. Constas et al. (2013); or Ansah et al. (2019) for a recent review -, no strong consensus has yet emerged on how to measure (and, a fortiori, to assess) resilience interventions. There is an urgent need to develop more rigorous methods to measure and assess resilience interventions in the field (Béné 2013). In this paper we present one of the first quantitative evaluations of resilience interventions based on a Difference in Difference design. The program to which this evaluation was applied, is the Enhanced Coastal Fisheries (ECOFISH) program implemented by the Government of Bangladesh with the collaboration of the WorldFish Center and financial support from USAID. The program was implemented in the southern part of Bangladesh with the objective to strengthen the resilience of the targeted communities.

Bangladesh, one of the most disaster-prone countries in the world, experiences a high number of weather-related adverse events every year. The majority of those events are related to the Southwest monsoon and the two cyclone seasons which affect the country on an annual basis. Those adverse events include cyclones, torrential rain, landslides, flash floods, storm surges, salinity intrusion, and extreme temperature (Chowdhury 2002; Alam 2004). In total 30 to 50 percent of Bangladesh is estimated to be affected by severe climatic or environmental shocks every year (Ahmed 2006; Dasgupta et al. 2010).

In the rural areas affected by these events, the population depends for a large part on natural resources, during normal times but also during times of crisis. Fisheries (inland and coastal) are a central part of the livelihood of many of those people, as a source of incomes as well as a source of food (Kleih et al. 2003; Islam et al. 2016; Mohammed et al. 2016). The Hilsa fishery, for instance, which target hilsa shad, the national fish of Bangladesh, is estimated to support more than half a million people directly involved in fishing activities, as well as around four to five million people involved with the trade of that fish (Halder 2004; BOBLME 2012; Mohammed and Wahab 2013; Mohammed et al. 2016).

In this context the aim of the ECOFISH program was to improve the status of the hilsa fisheries as a way to strengthen the resilience of these populations and in particular help them secure or improve their income and food security in the face of environmental or socio-economic shocks. More formally, the objective of the program was “to improve household and community resilience by backing up the livelihoods and coping capacities of the marginalised poor, including women, and reducing their risk to ecosystem threats, including impacts of climate change” (ECOFISH Annual Report 2018, p.10). The ECOFISH program was completed in 2019 and an impact evaluation exercise had been included in the workplan to assess the effectiveness of the program.

General Context of the Evaluation

ECOFISH

Enhanced Coastal Fisheries in Bangladesh (ECOFISH-Bangladesh) was a five-year (2014–2019) initiative funded by the United States Agency for International Development (USAID) and implemented jointly by WorldFish -a member of the Consultative Group for International Agricultural Research (CGIAR)- and the Department of Fisheries (DoF) of the Ministry of Fisheries & Livestock (MoFL). The project supported the DOF and local communities to establish a collaborative science-based “co-management” that focuses on the reduction of overfishing, protection of hilsa juveniles and brood in the peak spawning season, through the involvement of stakeholders in 12 coastal districts, with a target of supporting 20,000 fisher households.

Co-management Structures

Hilsa fishery of Bangladesh is characterised by complex institutional and governance factors. These include both formal and informal institutional arrangements, and multiple stakeholders including public, private and civil society sectors, and involving a range of linkages across sectors and areas of responsibilities. The community supports have been ensured through organizing groups and promoting community empowerment focusing on women and their livelihood diversification. The project established Hilsa Conservation Groups (HCGs), Hilsa Ghat (landing centre) Groups (HGGs), fisher women’ Community Savings Groups (CSGs), and Community Fish Guards (CFGs) in locations close to sanctuaries, breeding grounds and hilsa migration routes. These community groups acted as building blocks for the deployment of adaptive co-management approaches, including awareness building programs and livelihood support to fisher households. Members of these groups received extensive training to enhance their understanding of the importance of sustainable management of hilsa for their livelihoods and wellbeing. This is believed to be effective in building and strengthening local institutions and enhancing the social capital of the fisher communities, which is the primary asset of hilsa fishing dependent communities in Bangladesh.

Fishing Management Including Spatial and Temporal Closures

The Government of Bangladesh (GoB) has been implementing three types of fishing bans: (i) a fishing ban that lasts for 8 months (November-June) for fish smaller than 25 cm in total length; (ii) a Sanctuary fishing ban that prevent all type of fishing in the six Hilsa sanctuaries, and lasts for two months (March–April) in five sanctuaries and three months (November-January) in one sanctuary; and (iii) a brood hilsa fishing ban that lasts 22 days in the peak spawning period. The brood hilsa ban period and the sanctuary fishing ban periods are the most effectively reinforced fishing bans. Other management measures included prohibition of fishing by monofilament gill nets (current jal), set bag nets (behundi jal) and other illegal gear. During ban periods, access to Hilsa shad is reduced—hindering the production for food and income. For this reason a) compensation packages were designed and b) supplementary livelihoods strategies were provided.

Supplementary and Alternative Livelihoods Strategies and Compensation

Livelihood support activities were also included to address both the lack of assets and lack of skills underlying poor wellbeing outcomes. Targeted households received a choice of productive assets aimed to assist them in their basic needs fulfillment, generate alternative and supplementary income, and enhance their productivity. This was accompanied by skills training specific to the assets provided. Moreover, women enrolled in CSGs were also provided with business literacy training and access to micro-finance, along with ensuring access to easy and interest free soft loans (only service charge provided) to reduce dependency on high-interest loans from external non-formal credit providers.

Conceptual Framework

In this section we present the conceptual framework used to structure the evaluation. For this, we build on some recent advances made in the conceptualization of resilience measurement in the context of humanitarian and food security interventions (see e.g. von Grebmer et al. 2013; Constas et al. 2013, 2014). One of the key principles that underlie those different works is the recognition that resilience should not be seen as the final goal of a development intervention but instead as an intermediate outcome leading toward the achievement of a longer-term goal. This longer-term goal would typically be the improvement (or at least the maintaining of) people’s well-being -generally measured through their food security, health/nutrition status, or poverty level—in the face of shocks or stressors (Constas et al. 2014). A second important principle that emerges from these works is that resilience in the context of humanitarian or development interventions is mainly about people’s capacities, at individuals, households, communities, or institutions level, to adequately deal with shocks and stressorsFootnote 1 (Constas et al. 2013; Béné et al. 2014).

Theory of Change and Impact Pathway

Framed into a theory of change, this understanding of resilience means that the activities of a resilience intervention should be designed to strengthen the absorptive, adaptive, and transformative capacities of the target population (Frankenberger et al. 2012; Béné et al. 2014; Grist et al. 2014). This direct outcome should then help the targeted households adopt more appropriate responses in the face of shocks and stressors. By “appropriate” responses we mean responses that are less likely to induce long-term detrimental outcomes, in line with the definition of resilience proposed by Constas et al. (2013, p. 6) “Resilience is the capacity that ensures adverse stressors and shocks do not have long-lasting adverse development consequences”. Of particular concern in this regard are the short-term coping responses such as reducing health expenses or food consumption, selling productive assets, or engaging in non-sustainable activities (e.g. over-exploiting natural resources) which have been shown to generally lead to detrimental outcomes (Devereux 1993; Dercon et al. 2005; Coates et al. 2006). Instead, the objective of resilience initiatives is to help households engage in more positive responses such as adaptive or transformative strategies (e.g. execution of preparedness plans, adoption of climate smart practices). The adoption of those more appropriate responses as an intermediate outcome is then expected to lead to the improvement of the target population’s resilience per se (understood as the ability of these populations to handle shocks more appropriately). This level of resilience can be measured by assessing the beneficiaries’ effective recovery (the ‘bouncing back better’ element of resilience as often referred to in the literature).Footnote 2 This improved resilience is then expected to contribute to the programme’s ultimate goal—that is, to improve the long-term well-being of the beneficiaries of the intervention. Those series of successive changes are shown in Fig. 1 in the form of a generic theory of change (ToC) adapted to the case of the ECOFISH program.

Fig. 1
figure 1

Generic theory of change (ToC) of a resilience intervention—adapted here to the ECOFISH program. See text for the details of the different steps constituting the ToC and the subsequent evaluation hypotheses (Hypothesis 1,2, and 3 as indicated in the diagram)

Full size image

Building on this theory of change, the impact pathway of the ECOFISH project was elaborated (see Fig. 2). Along this pathway, the intermediate outcome, ultimate outcome and long-term impact reflect the last three steps of the theory of change: the types of responses adopted by the households, along with the immediate effects of the shock/stressors are the main factors that determine the ability of these households to handle positively (or not so positively) the shocks/stressors and eventually determine the longer term impact, measured here through changes in food and nutrition security of the households.

Fig. 2
figure 2

Overall resilience impact pathway of the ECOFISH project

Full size image

The left hand side of the impact pathway as shown in Fig. 2 diverges slightly from the ToC and may therefore deserve more explanation. It shows that the interventions of the program (alternative income generating activities (AIGAs), Hilsa Conservation Groups (HCGs), etc.) are theorised to build the household and community resilience capacities through two main paths. The first corresponds to the ‘conventional’ path, as widely described in the resilience literature—and in the ToC above- whereby project activities are expected to strengthen the tangible/directly measurable determinants of resilience (such as levels of income, assets, livelihood strategies, knowledge or access to infrastructure and services) at the individual, household or community level. A rich and growing body of literature has already discussed these factors (see, e.g., Alinovi et al. 2010; Pelletier et al. 2016; d’Errico et al. 2018). The second path builds on a more recent part of this literature which highlights that, in addition to those conventional elements, individual and household responses to shock/stressors (or even possibly community responses) are also influenced by psycho-social factors such as people’s aspiration, individual or collective level risk aversion, or sense of self-efficacy (Jones and Tanner 2015; Clare et al. 2018; Béné et al. 2019).

The final component of the determinant of resilience shown in Fig. 2 relates to the wider ecosystem health. Since poor and even less poor households living in rural areas are known to depend heavily on natural resources for their livelihoods (e.g. Beck and Nesmith 2001), the ECOFISH activities that were aimed at restoring or improving the general ecosystem health—and in particular the conditions of the hilsa fisheries resources- were also expected to contribute to build people’s resilience capacities.

Altogether those different path are assumed to constitute the main ways the ECOFISH would contribute to build the resilience capacities of the beneficiaries of the program.

Evaluation Hypotheses

Based on the impact pathways described above, three assessment hypotheses were identified; one at the intermediate outcome level, one at the final outcome level and one at the ultimate impact level. These hypotheses were then formulated to follow a treatment vs control design.

Hypothesis 1 (at the intermediate outcome level)

If the ECOFISH project activities have been effective in building beneficiary households’ resilience capacities, the types and/or frequencies of response(s) put in place by these beneficiaries should differ from those adopted by control households under the same circumstances. In particular:

Hypothesis (1)

  • ECOFISH beneficiaries show lower propensity to adopt negative coping strategies than households in the control group

  • ECOFISH beneficiaries show higher propensity to adopt positive (adaptive/transformative) responses than the control group

Hypothesis 2 (at the final outcome level)

The more appropriate responses adopted by the beneficiaries (Hypothesis 1) are expected to lead to stronger resilience (higher ability to ‘bounce back’) in the face of adverse events. More specifically:

Hypothesis (2)

  • ECOFISH beneficiaries show higher rate of recovery in the face of adverse events than households in the control group (everything else being equal)

Hypothesis 3 (at the long-term impact level)

The stronger ability to ‘bounce back’ expected to characterise beneficiaries in the face of adverse events (Hypothesis 2) should translate further along the impact pathway into higher long-term well-being indicators. More specifically:

Hypothesis (3)

  • ECOFISH beneficiaries show higher levels of wellbeing than households in the control group (everything else being equal)

Note that those three hypotheses correspond to specific steps along the impact pathway / ToC. As such they are also shown on Fig. 1. Note also that although resilience is acknowledged to be a concept relevant at several levels (from individual to institution) the unit of analysis in this evaluation framework is the household. The reason for this is that the unit of intervention of most development projects (including here the ECOFISH program) is usually the households. The three assessment hypotheses were therefore formulated to test the theory of change at this level.

Difference in Difference and Generic Models

A baseline and an endline surveys were completed as part of the Monitoring and Evaluation plan of the program (MEL Plan 2015, pp. 18–20), with household panel data collected pre- and post-intervention, thus allowing us to structure the evaluation framework around a difference-in-difference (DiD) design. This means that the evaluation was set to compare treatment and control groups before (baseline) and after (endline) the program.

As shown in Ravallion (2008) and others, the DiD estimate can be calculated using a regression framework. In particular, the estimating equation could be specified using a two-period OLS model:

$$Y_{it} = \alpha + DD \cdot T_{i} t + \beta T_{i} + \delta t + \varepsilon_{it}$$
(1)

where T is the treatment dummy (T = 0 for control, T = 1 for treatment) for household i, t is the time dummy (t = 0 for baseline; t = 1 for endline), and DD is the coefficient of the interaction of T and t. It can be shown that this interaction DD represents the DiD estimate of the impact of the program on the outcome Y assuming that the unobserved heterogeneity in the model is time invariant and uncorrelated with the treatment over time (Ravallion 2008).

The OLS model (1) can then be “enhanced” in two ways:

  1. A.

    Other covariates (in particular those characterizing the household socio-economic and demographic status) that may affect the outcome of the program should be controlled for. To this purpose, regression (1) can be extended by including those other covariates and get the net effect of program on the outcome of interest:

    $$Y_{it} = \alpha + DD \cdot T_{i} t + \beta T_{i} + \delta t + \gamma V_{it} + \varepsilon_{it}$$
    (2)

    where \(V_{it}\) is the household covariates to be controlled for.

  2. B.

    The adoption of a fixed-effects regression (OLS-FE) instead of the extended OLS model (2) where the fixed-effect \(\mu\) is introduced to control for household’s time-invariant characteristics that may also influence the outcome variable.

    $$Y_{it} = \alpha + DD \cdot T_{i} t + \beta T_{i} + \delta t + \gamma V_{it} + \mu_{i} + \varepsilon_{it}$$
    (3)

Combining A and B together allows us not only to account for the unobserved time-invariant heterogeneity but also for heterogeneity in observed characteristics over the time setting.

Structure of the Questionnaire

The structure and the wording of the questionnaire were kept similar between baseline and endline surveys. The questionnaire was designed to capture the specific information necessary for the measurement and evaluation of resilience in line with the impact pathways described above. In particular the following modules were included:


Determinants of resilience:

  • Household social and demographic characteristics, including: household composition; age, gender and education of the household head; nature and market values of home and productive assets; levels of saving and debts; main livelihood and sources of income.

  • External services/support received by households in time of crisis—including both governmental and non-governmental agencies;

  • Level of bonding social capital (in the form of household personal social network and community social cohesion);

  • Psycho-social determinants of (subjective) resilience including: self-efficacy, optimism; and self-confidence

  • Ecosystem and resource status: self-assessed catch level and income derived from hilsa and other fish resources;

Shock and stressors:

  • Inventory (nature, frequency and severity) of the different covariant and idiosyncratic shocks/stressors that have affected any member of the surveyed households in the 36-month period prior to the survey;

Intermediate outcomes:

  • Type(s) of response(s) adopted by households following (or in anticipation of) the occurrence of specific shocks and stressors. The responses included negative coping strategies (namely: reducing household consumption, changing type of food consumed, reducing expenses, selling productive assets, or borrowing money) as well more positive adaptive or transformative strategies including execution of preparedness plans for the house and adoption of new activities in and outside the fisheries (namely: save money; store water and prepare dry or cooked food in advance; strengthened house roofing structure, increase information about early warning systems (EWS); livelihood diversification (non-fishing activities, investment in small livestock, home-gardening), collective work to reinforce the embankment prior to the flood season);

Ultimate outcome:

  • Resilience index: household self-assessed recovery rate from the different shocks/stressors experienced in the 36-month period prior to the survey -see more details below for the computation of this indicator;

Long-term impact:

  • Household Food Insecurity Access scale (HFIAS) (Coates et al. 2007);

  • Household Food Consumption Score (FCS) (WFP 2007) and

  • Household Dietary Diversity Scores (HDDS) (Swindale and Bilinsky 2006) -see more details below for those three indicators.

Those different variables have been summarized in Table 1

Table 1 The various variables included in the resilience analysis
Full size table

.

Resilience Index

In theory, measuring resilience would require high frequency data documenting the instantaneous change in a household’s well-being, welfare (income, assets), food security or nutrition indicators in the days, weeks and months following a specific shock (Barrett and Headey 2014; Béné et al. 2015). Since such high frequency information was not available in the case of the ECOFISH program (and is actually very rarely available for any development program), we based the evaluation on a less ‘data-hungry’ resilience indicator. For this, we adapted an approach developed initially by Béné et al. (2016, 2017) in similar food security and humanitarian crises interventions. Under this approach, resilience outcomes are assessed using psychometric techniques (self-reporting evaluation based on Likert scale) whereby households assess how they recovered from the various past adverse events they have experienced. The self-assessment process is shock specific and based on two distinct recall questions: (i) self-recovery from past events; and (ii) self-recovery compared with the rest of the community. For each question, respondents are asked to select answers from a 6-point Likert scale system for the first recall question, and a 5-point Likert scale system for the second recall question. The details of the questions and associated scoring are provided in Table S1 in Supplementary Materials. The index is then constructed by combining the answers at the household level, producing a self-assessed level of resilience (or ability to recover) which is specific to shocks/stressors -see formula (1).

$$RI_{h}^{s} = {}_{1}^{6} Q_{h}^{{\left( {i,s} \right)}} \cdot {}_{1}^{5} Q_{h}^{{\left( {ii,s} \right)}}$$
(4)

where \(RI_{h}^{s}\) is the resilience index of household (h) related to adverse event (s); and \({}_{1}^{6} Q_{h}^{{\left( {i,s} \right)}}\) and \({}_{1}^{5} Q_{h}^{{\left( {ii,s} \right)}}\) are the respective self-assessed scores of household (h) to the two recall questions (i) and (ii). The component \({}_{1}^{6} Q_{h}^{{\left( {i,s} \right)}}\) varies from 1 to 6, while \({}_{1}^{5} Q_{h}^{{\left( {ii,s} \right)}}\) varies from 1 to 5; consequently \(RI_{h}^{s}\) varies from 1 to 30, with 1 indicating a very poor level of resilience, while 30 would indicate a full recovery.

Well-being Index

The long-term impact of the ECOFISH on its beneficiaries was assessed through the evaluation of potential change in three indicators:

  1. I.

    Household Food Insecurity Access Scale (HFIAS). The HFIAS is an index constructed from the responses to nine questions regarding people’s experiences of food insecurity in the previous four weeks. It measures household access to sufficient food of adequate quality and percentage of households that are food secure. A score is then calculated based on these frequency responses—see details in Coates et al. (2007). The inverse of the score was used here so that the measure increases with increasing household food security.

  2. II.

    Household Dietary Diversity Score (HDDS). The HDDS reflects the quality of households’ diets by accounting for the total number of food groups, out of 12, from which household members consumed food in the last 24 h before the survey. The HDDS is then tabulated by calculating the average number of groups consumed amongst all the members of the household -see Swindale and Bilinsky (2006) for more details. The value ranges from 0 to 12.

  3. III.

    The Food Consumption Score (FCS) is an index that aggregates household-level data on diversity and frequency of food groups consumed during the last seven days. As such it is a composite score based on dietary diversity, food frequency, and relative nutritional importance of different food groups. Respondents are asked about frequency of consumption (in days) of food items grouped into 8 standard food groups over a recall period of past 7 days. The consumption frequency of each food group is then multiplied by an assigned weight that is based on its nutrient content—see WFP (2007) for more details. Those values are then summed up, obtaining the Food Consumption Score (FCS).

Methods and Data

Sampling Frame

The framework developed for this impact evaluation follows an approach based on a quasi-experimental design, applying a DiD protocol. The sampling covered 1,200 households including 600 beneficiaries (treatment) (T) and 600 control households (C). The two groups were sampled in panel through the baseline and the endline surveys. The overall sampling was structured as a two-stage cluster sampling based on a sample frame generated by a separate household listing exercise. At the first stage, a sample cluster was selected independently with probability proportional to the cluster’s population in each stratum. The strata were the six districts in Bangladesh encompassing the program area—Chandpur, Bhola, Laxmipur, Barisal, Jhalokathi and Pirojpur. The second stage involved the use of systematic random sampling at village level to select a set number of households (50 in each village) to be included in the evaluation.

In order to reduce the heterogeneity within and between groups (treatment and control) which is likely to result from the relatively large geographical area covered by the program, the sampling effort focused on the central region of intervention of the program (see Fig. 3). The control villages were then selected in the same districts than the treatment villages in order to optimize the comparability between the two groups, but in different unions, in order to reduce the risk of 'spillover effect'.Footnote 3

Fig. 3
figure 3

Geographic location of the villages where the baseline/endline surveys were administered within the 6 districts where the ECOFISH programme operated

Full size image

Data Collection

The field work took place in Aug-Sept 2016 for the baseline and June-Jul 2019 for the endline. The data collection was conducted by 24 enumerators, under the supervision of four field supervisors and the ECOFISH monitoring, evaluation and learning leader, after that the enumerators and field supervisors had been provided with a three-day training to familiarise themselves with the questionnaire and the objective of the survey. Data collection was carried out through face-to-face interviews with the head of household. Android tablets (Google Nexus) were used for data recording, using ODK (Open Data Kit) software. The use of mobile devices and an electronic questionnaire allowed for the integration of data validation rules and consistency checks as part of data collection. It also reduced data entry burden, as data were entered at the interviewer level. Every record was stored and uploaded to a cloud server utilizing the built-in internet connectivity of the devices. Descriptive analysis and regression models were computed using Stata v.13.2.

Results

Amongst the 1200 households initially surveyed at baseline, 1185 were found and surveyed again at endline (597 in the treatment group, and 588 in the control group), with an acceptable rate of attrition of 1.25%.

Comparability Between Control and Treatment Households

The baseline data from both groups (Treatment and Control) appeared broadly comparable in terms of social-ecological, livelihood, and demographic characteristics (determinants of resilience), as well as exposure to adverse events.

One of the critical requirement in building a proper counterfactual in the context of resilience interventions is to ensure that control and treatment households at baseline are as comparable as possible with regard to their general (social, economic, demographic) characteristics as well as their exposure to shocks and stressors. The data reveals that as for as age (t = -0.899, p = 0.37); illiteracy level (F = 0.013, p = 0.91); household size (t = -1.892, p = 0.06); livelihood diversification (t = -1.269, p = 0.20); level of assets (t = -1.495, p = 0.13) and savings (t = -1.508, p = 0.13); personal social network (t = -1.757, p = 0.08); and psycho-social index (t = -0.075, p = 0.94), the control and treatment groups were statistically comparable.

The analysis also shows that the level of exposure to covariate and idiosyncratic shocks and stressors does not diverge noticeably between the two groups. The beneficiaries of the ECOFISH (treatment) reported to have been affected on average by 5.07 adverse events in the 36 months before the survey while the control households reported on average 4.90 for the same period. As far as the nature of those events is concerned, the data suggest a relatively similar ‘shock/stressor profile’ between the two groups (Fig. 4). In particular, the five most reported shocks/stressors across households were the same ones: two-month fish ban (reported by 99% of the households in both treatment T and control C); serious illness of one (or more) members of the household (65% T; 71% C), loss of assets (57% T; 53% C); fish catch slow decline (46% T; 41% C), and water-logging (31% T; 36% C).

Fig. 4
figure 4

Shocks/stressor profiles showing the list of adverse events that affected the treatment and control households in the 36 months prior to the baseline survey. Only the 12 most reported events are represented in the diagram

Full size image

Finally, the household catch and income derived from hilsa and non-hilsa fishing activities were also compared between the two groups at baseline with non-statistical difference for hilsa catch (t = 0.70, p = 0.46); non-hilsa catch (t = -0.13, p = 0.89); hilsa income (t = -0.73, p = 0.46); non hilsa income (t = 0.21, p = 0.83).

In the rest of this section we now present the results of the DiD tests conducted at the different levels of the impacts pathway.

Assessing the Effect of ECOFISH at Intermediate Outcome Level

Our first evaluation hypothesis was that if the activities of ECOFISH have been effective, the propensity of the beneficiaries of the programme to engage in (negative and positive) responses should differ from those adopted by control households. We tested those two hypotheses separately (first the negative coping strategies, and second the positive adaptive/transformative strategies), using OLS-FE models while controlling for household covariates.

Because household responses are shock-specific, we also needed to account for the nature of the event, especially for the analysis of the negative coping strategies. We therefore focused on the three most frequently reported events, namely the 2-month fishing ban; serious illness; and loss of assets. For each of those three major events we identified the households who declared having been affected by those shocks at both baseline and endline and then analyse their coping strategies.

The results shows that the propensity to engage in negative copying strategies decrease between the baseline and the endline only for the most reported event (two-month fishing ban), but not for the two other main shocks (serious illness and loss of assets) (Table S2 in Supplementary Materials). The OLS-FE models testing the DiD are also non-conclusive (Table 2). The variable DD is statistically significant for the two-month fishing ban and serious illness shocks, but in the “wrong way”, in the sense that the propensity to engage in negative coping strategies is lower in the control group than in the treatment group when comparing baseline and endline. For the third major shock (loss of assets), the DD variable is not significant. Overall, this means that the empirical results do not confirm our first hypothesis for negative coping strategies: the ECOFISH interventions did not reduce significantly the propensity of households to engage in negative coping strategies.

Table 2 Testing the effect of ECOFISH on coping strategies, using fixed effects models while controlling for household covariates. Test completed for the three main shocks: two-month fishing ban (model1); serious illness affecting one (or more) members of the household (model2); and loss of assets (model3), and the propensity to engage in coping strategies as dependent variable
Full size table

For positive responses, results are more conclusive. The propensity to engage in adaptive or transformative responsesFootnote 4 increases between baseline and endline and is also higher amongst treatment households than control households at endline (Table S3 in Supplementary Material). The OLS-FE model shows that the variable DD is statistically significant (Table 3—left), indicating that the ECOFISH project has been effective at strengthening the capacities of the households to engage in positive responses when affected by adverse events.

Table 3 Testing the effect of ECOFISH on household adaptive responses (Posit_responses, left) and on their capacities to recover from shock (Resi_index RI, right), using fixed effects models while controlling for household covariates
Full size table

Assessing the Effect of ECOFISH at Final Outcome Level

Our second evaluation hypothesis (Hypothesis 2) states that the adoption of more appropriate responses by the beneficiaries of the program should help strengthening the capacity of those beneficiaries to recover from adverse events. This capacity to ‘bounce back’ was estimated through the resilience index computed at household level (see Sect. 3.4).

Descriptive statistics reveal that this resilience index increases between baseline and endline and is also higher amongst the treatment group than amongst the control group (Table S4 in Supplementary Material). The estimation of the OLS-FE model (Table 3—right) shows that the DD variable is statistically significant, thus confirming our Hypothesis 2: the ECOFISH project has been effective at strengthening the capacities of the households to recover from adverse events.

Assessing the Effect of ECOFISH at Impact Level

Our third evaluation hypothesis (Hypothesis 3) focused on the potential long-term impact of the project. It was assumed that if the capacity of households to bounce back after being hit by an adverse event has been enhanced by the project (our Hypothesis 2 above), this more effective recovery should help those households to increase their general well-being, in particular when compared to households who have been hit by the same shocks but did not benefit from the ECOFISH program.

The descriptive statistics (Table S5 in Supplementary Materials) indicates that the three indicators of wellbeing evolve in the expected direction over time: both HDDS and FCS increase while HFIAS decreases. The data also shows that the HDDS and FCS of the treatment groups are higher at endline than those of the control groups, while the treatment group’s HFIAS is lower than that of the control group. The OLS-FE models reveals, however, that the DD variable is statistically significant for the HDDS, but not for the HFIAS or the FCS (Table 4). In other terms, Hypothesis 3 is confirmed only for HDDS.

Table 4 Testing the long-term impact of ECOFISH on household diet diversity score (HDDS), household food insecurity access scale (HFIAS); and household food consumption score (FCS), using fixed effects models
Full size table

Discussion and Concluding Remarks

While the number of development projects that aim at strengthening the resilience of vulnerable populations in low or middle income countries is still increasing (Béné et al. 2014), the frameworks available to assess rigorously the impacts of those resilience-focused interventions are still underdeveloped (Ansah et al. 2019). This defect is partially explained by the well-known difficulties related to the evaluation of on-the-ground development interventions—in particular the issue of selection bias and the problem of absence of good counterfactual (De Janvry et al. 2011; Gertler et al. 2016). Those technical issues are further complicated in the case of resilience interventions by the fact that resilience is itself a latent variable which, by nature, is not directly measurable and therefore difficult to assess (Béné 2013; d’Errico et al. 2016).

In this paper we assessed the impact of the Enhanced Coastal Fisheries (ECOFISH) project. The main objective of the project, which was implemented from 2016 to 2019 in Bangladesh, was to strengthen the resilience of the local communities who depend for a large part on fishing to sustain their livelihood. As such, ECOFISH falls into the category of projects discussed above: difficult to assess due of its natural experiment nature and complicated by its focus on resilience.

Using the impact pathways of the project, we were able to structure the evaluation around three distinct hypotheses in line with the project’s theory of change. We then tested those hypotheses using a difference-in-difference (DiD) framework.Footnote 5 The first hypothesis was subdivided into two parts; one looking at negative copying strategies that have long been documented to have detrimental implications for the welfare, food security and nutrition status of households (see e.g. Devereux 1993; Dercon et al. 2005; Coates et al. 2006). We hypothesized that if the ECOFISH project has been effective, beneficiaries should be less likely to engage in those negative strategies than households in the control group. The DiD test was therefore set up to test this hypothesis for the three most reported adverse events (fishing ban, serious illness and loss of assets). The second part of the hypothesis looked at the adoption of more ‘positive’ responses, and assumed that beneficiary households should show a higher propensity to engage in those positive response than the control group.

Testing the two elements of this first hypothesis shows mixed results. While the second part was verified -confirming that households who benefitted from the ECOFISH activities have a higher propensity to adopt adaptive/transformative responses than non-beneficiaries at endline-, for the first part (adoption of negative coping strategies), the test did not confirm the hypothesis for neither of the three adverse events tested—suggesting that the project was not successful at building the capacities of the households to escape the vicious circle of detrimental/harmful strategies.

While cautiously avoiding to over-interpret these results, the concurrence of the success of the project in encouraging households to adopt positive responses alongside its failure to prevent those same households from engaging in negative strategies suggests a ‘dissymmetry’ in the ability of the ECOFISH project to build people’s resilience. It seems ‘easier’ for a development project like ECHOFISH to strengthen the positive (adaptive) resilience of people than to eliminate the more negative (absorptive) component. Disturbingly, this dissymmetrical effect was also observed in another recently evaluated resilience project in Niger (Béné et al. 2019). This observation also highlights the importance, when one evaluates resilience projects, of not simply testing the changes in adoption rate of positive responses amongst the targeted population but also the changes in the rate of adoption of the more negative strategies. Unfortunately, in many instances, projects seem to focus more often on the positive aspects (adoption of adaptive responses) (see e.g. Brownhill et al. 2014; Ahmed et al. 2016; CARE 2017) and ignore or overlook the possible occurrence of negative/detrimental strategies. This bias may be related to the fact that resilience is often conflated with adaptive capacities (Gallopin 2006). While adaptive capacity is indeed an important element contributing to the construction of resilience, it is now well established that this is only one dimension and that other important processes—such as absorptive capacity or anticipation- need also to be considered (Constas et al. 2014; Béné et al. 2014).

In the case of the ECOFISH project, if the theory of change of the project is correct, the increased propensity of beneficiary households to engage in adaptive/transformative responses could also be interpreted as the causal factor explaining the positive result observed at the final outcome level under our second evaluation hypothesis; namely that those beneficiaries report now a higher recovery rate than the households in the control group.

This paper is not the first one which uses recovery path to assess the resilience of population (either individual, households or communities). In the humanitarian literature, using recovery of households or communities affected by natural disasters to evaluate their resilience is common—see e.g. Zhang and Peacock (2009); Chang (2010); Bevington et al. (2011); Thornley et al. (2014); Jordan and Javernick-Will (2015), amongst others. But fewer of those studies used this recovery path for assessing resilience changes after an intervention. Knippenberg and Hoddinott (2017) is one of them. These authors, after assessing the recovery path of the beneficiaries of the Productive Safety Net Program (PSNP) in Ethiopia through an instrumental variable estimator, showed that receipt of PSNP payments reduced substantially the initial impact of drought shocks and eliminates their adverse impact on households’ food security within two years. In Bangladesh, Béné et al. (2017) developed an ex-post treatment vs. control evaluation framework to assess the impact of the Enhancing Resilience (ER) program implemented by WFP. They found that the ER program has been successful at improving the recovery of the ER beneficiaries. In the present case, using a DiD test implemented through the estimation of an OLS-FE, we were able to show that the ECOFISH intervention successfully increased the recovery of the beneficiaries of the project.

The third and last hypothesis that structured this evaluation looked at the long-term wellbeing of the population, assuming that the beneficiaries of the ECOFISH should have been able to improve (or at least better protect) their wellbeing status than household who did not benefit from the activities of the project. Three different indicators of long-term wellbeing were considered (HFIAS, HDDS, and FCS), and for each of them, the evaluation hypothesis was tested using an OLS-FE model. While the descriptive statistics showed that the three indicators move in the expected directions (HDDS and FCS increased while HFIAS decreased), the OLS-FE models reveals that the changes are statistically significant for only one of the indicators (the HDDS), suggesting that the ECOFISH has not been totally successful at improving the wellbeing of its beneficiaries.

This last result may sound surprising—or disappointing—given that households in the treatment group were showing higher levels of recovery than in the control group. One possible explanation is that despite these higher levels of recovery, the time elapsed since the start of the project (3 years) has been too short for its positive outcomes to transform into significant impact on the long-term well-being of the beneficiaries. This suggests that it takes time not only to build people’s resilience (Pelletier et al. 2016) but also to see the impact of this resilience on people’s well-being.