Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.789877986s FAIL (2025-07-16 10:02:28 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 247393737 (552h49m37.018035092s) [noise=true , address=45.91.116.5:4298 , piece_id=SO5TK54HNRJYPBWWO7G5Z7LILWBVYRO7GYSUBLGXP3AENSNNWUHQ, seg_pos={0 0}, upload=442102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 16 10:00:02.247425508 (552h49m37.018066853s) [seg_pos={0 0}, upload=442102]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 16 10:00:02.247519749 (552h49m37.018161094s) [upload=442102]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 10:00:02.247526431 (552h49m37.018167776s) [upload=442102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 10:00:02.567193618 (552h49m37.337834964s) [upload=442102]: Done uploading segments... Jul 16 10:00:02.567223626 (552h49m37.337864971s) [upload=442102]: Upload finished. err: <nil> Jul 16 10:00:35.322798409 (552h50m10.093439744s) [upload=442109]: Starting upload Jul 16 10:00:35.322854246 (552h50m10.093495581s) [upload=442109]: Done starting upload Jul 16 10:00:35.322959147 (552h50m10.093600482s) [upload=442109]: Uploading segments... Jul 16 10:00:35.323170992 (552h50m10.093812327s) [upload=442109]: Got next segment. Inline: false Jul 16 10:00:35.323212882 (552h50m10.093854217s) [upload=442109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 10:00:35.323218092 (552h50m10.093859427s) [upload=442109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 10:00:35.563095085 (552h50m10.33373642s) [seg_pos={0 0}, upload=442109]: Begin upload segment... Jul 16 10:00:35.563136655 (552h50m10.33377799s) [seg_pos={0 0}, upload=442109]: Done begin upload segment. Jul 16 10:00:35.563158777 (552h50m10.333800112s) [noise=true , address=45.91.116.10:4082 , piece_id=5SQCN7SRWSQWDEUD6CWGKXPIRJZ3AUTMRE3TWOSIE5LB6AHXI56Q, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.563198944 (552h50m10.333840279s) [noise=true , address=104.36.82.9:4005 , piece_id=VRB6S4DFGLIOXJUK3HODA7F3MNEGWCJKLEQ75TUFEPZW334UITVQ, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.563587467 (552h50m10.334228802s) [noise=true , address=77.90.31.18:4806 , piece_id=XUQS52FOQ3QI3V3VXZ2T3OJIML4TFADNTK4IUR4QZMHXOFOGXI7A, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.563707898 (552h50m10.334349233s) [noise=true , address=45.91.116.7:4011 , piece_id=FO6J6XC73ANPA23IMGP3D5K5MJVNTX7EBJT3ASYUWAJTSK3PLS2A, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.563853246 (552h50m10.334494581s) [noise=true , address=77.90.31.16:4735 , piece_id=HD3S7QZXO4RMPYCL23T3Z74YTJVG7C5MYBWKB244J7WKDNNXCCTA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564066364 (552h50m10.334707709s) [noise=true , address=45.91.116.14:4189 , piece_id=2EOPVJYRHRLZE57BPRX6AO7GAFSNOS5GDJ3ARZAICQUNTAMF3JOA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564126779 (552h50m10.334768114s) [noise=true , address=77.90.31.12:4589 , piece_id=PP4Y5NRJ6JVK4LE6BGTRDN4MCD5JBUY7REZ4BBVQ65YUVZ7UCHWA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564460948 (552h50m10.335102283s) [noise=true , address=104.36.82.3:4767 , piece_id=CCBSXDZKI6GJNPKY5N46KQOMOJQU6YGFH3NTRYKIGZJOCGV3FEBA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564516885 (552h50m10.33515822s) [noise=true , address=77.90.31.14:4661 , piece_id=RQUAB4AGZSNROLUFJEVE2F4YO67LJDYQKWQ4KXDZDUACMAZ36ZXA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564768236 (552h50m10.335409571s) [noise=true , address=45.91.116.12:4227 , piece_id=BUY5Z3M7E6QDKLHNIODRQK3RKJ32XPNXZG7EDKQF6IWFBRAPC67Q, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564775581 (552h50m10.335416906s) [noise=true , address=77.90.31.11:4553 , piece_id=ER33GQYXW5WQCSR7MTAFCS5LREVU4IEQFZVU25MIK27TOVZJSPMQ, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.565144586 (552h50m10.335785922s) [noise=true , address=77.90.31.13:4628 , piece_id=JFF7FVEJF4PI3EDCTMGCRWKRT536IKPR367EJDSU6NL5K432I2MQ, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.565412429 (552h50m10.336053794s) [noise=true , address=45.91.116.11:4118 , piece_id=6AJYPKWRE27Y5VYRHV643PH3MWLR7DQ3CJG7CRNGFUU5DXIPP6LA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.566888023 (552h50m10.337529358s) [noise=true , address=45.91.116.13:4154 , piece_id=7T7C7GQK6MMN2WB5K5GMT7MVIS5LDE7ZVJ3NJXUYSBROTVWED5QA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.567630944 (552h50m10.338272279s) [noise=true , address=77.90.31.10:4519 , piece_id=Y7FCGTWSZUYYQVVN4A2K5Y6HXYOYUBTDZ4BB3GKZ3CG36M4JG72Q, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.569125584 (552h50m10.339766919s) [noise=true , address=45.91.116.8:4047 , piece_id=PZOXUWLARQPBNSN67OX2ANYWWWOUXZ4BJTQAQRRDX3YWR7LUC7JQ, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:37.312772479 (552h50m12.083413814s) [upload=442109]: Next returned nil segment Jul 16 10:00:37.312790493 (552h50m12.083431828s) [upload=442109]: Waiting for error group managing segments... Jul 16 10:00:37.621481541 (552h50m12.392122876s) [noise=true , address=45.91.116.14:4189 , piece_id=2EOPVJYRHRLZE57BPRX6AO7GAFSNOS5GDJ3ARZAICQUNTAMF3JOA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.622538452 (552h50m12.393179787s) [noise=true , address=45.91.116.8:4047 , piece_id=PZOXUWLARQPBNSN67OX2ANYWWWOUXZ4BJTQAQRRDX3YWR7LUC7JQ, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.625336787 (552h50m12.395978122s) [noise=true , address=45.91.116.11:4118 , piece_id=6AJYPKWRE27Y5VYRHV643PH3MWLR7DQ3CJG7CRNGFUU5DXIPP6LA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.625486202 (552h50m12.396127537s) [noise=true , address=45.91.116.13:4154 , piece_id=7T7C7GQK6MMN2WB5K5GMT7MVIS5LDE7ZVJ3NJXUYSBROTVWED5QA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.643097378 (552h50m12.413738723s) [noise=true , address=45.91.116.12:4227 , piece_id=BUY5Z3M7E6QDKLHNIODRQK3RKJ32XPNXZG7EDKQF6IWFBRAPC67Q, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.646933618 (552h50m12.417574953s) [noise=true , address=77.90.31.11:4553 , piece_id=ER33GQYXW5WQCSR7MTAFCS5LREVU4IEQFZVU25MIK27TOVZJSPMQ, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.656912601 (552h50m12.427553936s) [noise=true , address=77.90.31.14:4661 , piece_id=RQUAB4AGZSNROLUFJEVE2F4YO67LJDYQKWQ4KXDZDUACMAZ36ZXA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.657051858 (552h50m12.427693193s) [noise=true , address=77.90.31.16:4735 , piece_id=HD3S7QZXO4RMPYCL23T3Z74YTJVG7C5MYBWKB244J7WKDNNXCCTA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.658382904 (552h50m12.429024239s) [noise=true , address=77.90.31.18:4806 , piece_id=XUQS52FOQ3QI3V3VXZ2T3OJIML4TFADNTK4IUR4QZMHXOFOGXI7A, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.659112881 (552h50m12.429754216s) [noise=true , address=77.90.31.12:4589 , piece_id=PP4Y5NRJ6JVK4LE6BGTRDN4MCD5JBUY7REZ4BBVQ65YUVZ7UCHWA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.662495022 (552h50m12.433136357s) [noise=true , address=77.90.31.10:4519 , piece_id=Y7FCGTWSZUYYQVVN4A2K5Y6HXYOYUBTDZ4BB3GKZ3CG36M4JG72Q, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.663463916 (552h50m12.434105251s) [noise=true , address=77.90.31.13:4628 , piece_id=JFF7FVEJF4PI3EDCTMGCRWKRT536IKPR367EJDSU6NL5K432I2MQ, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.695800476 (552h50m12.466441811s) [noise=true , address=45.91.116.10:4082 , piece_id=5SQCN7SRWSQWDEUD6CWGKXPIRJZ3AUTMRE3TWOSIE5LB6AHXI56Q, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.695821076 (552h50m12.466462411s) [seg_pos={0 0}, upload=442109]: Segment reached optimal threshold of 13 pieces. Jul 16 10:00:37.695931788 (552h50m12.466573133s) [noise=true , address=104.36.82.9:4005 , piece_id=VRB6S4DFGLIOXJUK3HODA7F3MNEGWCJKLEQ75TUFEPZW334UITVQ, seg_pos={0 0}, upload=442109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: expected piece hash; context canceled; EOF Jul 16 10:00:37.695964340 (552h50m12.466605675s) [noise=true , address=45.91.116.7:4011 , piece_id=FO6J6XC73ANPA23IMGP3D5K5MJVNTX7EBJT3ASYUWAJTSK3PLS2A, seg_pos={0 0}, upload=442109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Jul 16 10:00:37.696027892 (552h50m12.466669227s) [noise=true , address=104.36.82.3:4767 , piece_id=CCBSXDZKI6GJNPKY5N46KQOMOJQU6YGFH3NTRYKIGZJOCGV3FEBA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Jul 16 10:00:37.696084500 (552h50m12.466725835s) [seg_pos={0 0}, upload=442109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 16 10:00:37.696142571 (552h50m12.466783896s) [upload=442109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 10:00:37.696146709 (552h50m12.466788044s) [upload=442109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 10:00:38.117285014 (552h50m12.887926599s) [upload=442109]: Done uploading segments... Jul 16 10:00:38.117304631 (552h50m12.887945967s) [upload=442109]: Upload finished. err: <nil> Duration: 5.789909858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................................X] duration: 30.000470713s FAIL (2025-07-16 10:02:13 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 247393737 (552h49m37.018035092s) [noise=true , address=45.91.116.5:4298 , piece_id=SO5TK54HNRJYPBWWO7G5Z7LILWBVYRO7GYSUBLGXP3AENSNNWUHQ, seg_pos={0 0}, upload=442102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jul 16 10:00:02.247425508 (552h49m37.018066853s) [seg_pos={0 0}, upload=442102]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 16 10:00:02.247519749 (552h49m37.018161094s) [upload=442102]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 10:00:02.247526431 (552h49m37.018167776s) [upload=442102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 10:00:02.567193618 (552h49m37.337834964s) [upload=442102]: Done uploading segments... Jul 16 10:00:02.567223626 (552h49m37.337864971s) [upload=442102]: Upload finished. err: <nil> Jul 16 10:00:35.322798409 (552h50m10.093439744s) [upload=442109]: Starting upload Jul 16 10:00:35.322854246 (552h50m10.093495581s) [upload=442109]: Done starting upload Jul 16 10:00:35.322959147 (552h50m10.093600482s) [upload=442109]: Uploading segments... Jul 16 10:00:35.323170992 (552h50m10.093812327s) [upload=442109]: Got next segment. Inline: false Jul 16 10:00:35.323212882 (552h50m10.093854217s) [upload=442109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 10:00:35.323218092 (552h50m10.093859427s) [upload=442109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 10:00:35.563095085 (552h50m10.33373642s) [seg_pos={0 0}, upload=442109]: Begin upload segment... Jul 16 10:00:35.563136655 (552h50m10.33377799s) [seg_pos={0 0}, upload=442109]: Done begin upload segment. Jul 16 10:00:35.563158777 (552h50m10.333800112s) [noise=true , address=45.91.116.10:4082 , piece_id=5SQCN7SRWSQWDEUD6CWGKXPIRJZ3AUTMRE3TWOSIE5LB6AHXI56Q, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.563198944 (552h50m10.333840279s) [noise=true , address=104.36.82.9:4005 , piece_id=VRB6S4DFGLIOXJUK3HODA7F3MNEGWCJKLEQ75TUFEPZW334UITVQ, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.563587467 (552h50m10.334228802s) [noise=true , address=77.90.31.18:4806 , piece_id=XUQS52FOQ3QI3V3VXZ2T3OJIML4TFADNTK4IUR4QZMHXOFOGXI7A, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.563707898 (552h50m10.334349233s) [noise=true , address=45.91.116.7:4011 , piece_id=FO6J6XC73ANPA23IMGP3D5K5MJVNTX7EBJT3ASYUWAJTSK3PLS2A, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.563853246 (552h50m10.334494581s) [noise=true , address=77.90.31.16:4735 , piece_id=HD3S7QZXO4RMPYCL23T3Z74YTJVG7C5MYBWKB244J7WKDNNXCCTA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564066364 (552h50m10.334707709s) [noise=true , address=45.91.116.14:4189 , piece_id=2EOPVJYRHRLZE57BPRX6AO7GAFSNOS5GDJ3ARZAICQUNTAMF3JOA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564126779 (552h50m10.334768114s) [noise=true , address=77.90.31.12:4589 , piece_id=PP4Y5NRJ6JVK4LE6BGTRDN4MCD5JBUY7REZ4BBVQ65YUVZ7UCHWA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564460948 (552h50m10.335102283s) [noise=true , address=104.36.82.3:4767 , piece_id=CCBSXDZKI6GJNPKY5N46KQOMOJQU6YGFH3NTRYKIGZJOCGV3FEBA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564516885 (552h50m10.33515822s) [noise=true , address=77.90.31.14:4661 , piece_id=RQUAB4AGZSNROLUFJEVE2F4YO67LJDYQKWQ4KXDZDUACMAZ36ZXA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564768236 (552h50m10.335409571s) [noise=true , address=45.91.116.12:4227 , piece_id=BUY5Z3M7E6QDKLHNIODRQK3RKJ32XPNXZG7EDKQF6IWFBRAPC67Q, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.564775581 (552h50m10.335416906s) [noise=true , address=77.90.31.11:4553 , piece_id=ER33GQYXW5WQCSR7MTAFCS5LREVU4IEQFZVU25MIK27TOVZJSPMQ, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.565144586 (552h50m10.335785922s) [noise=true , address=77.90.31.13:4628 , piece_id=JFF7FVEJF4PI3EDCTMGCRWKRT536IKPR367EJDSU6NL5K432I2MQ, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.565412429 (552h50m10.336053794s) [noise=true , address=45.91.116.11:4118 , piece_id=6AJYPKWRE27Y5VYRHV643PH3MWLR7DQ3CJG7CRNGFUU5DXIPP6LA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.566888023 (552h50m10.337529358s) [noise=true , address=45.91.116.13:4154 , piece_id=7T7C7GQK6MMN2WB5K5GMT7MVIS5LDE7ZVJ3NJXUYSBROTVWED5QA, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.567630944 (552h50m10.338272279s) [noise=true , address=77.90.31.10:4519 , piece_id=Y7FCGTWSZUYYQVVN4A2K5Y6HXYOYUBTDZ4BB3GKZ3CG36M4JG72Q, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:35.569125584 (552h50m10.339766919s) [noise=true , address=45.91.116.8:4047 , piece_id=PZOXUWLARQPBNSN67OX2ANYWWWOUXZ4BJTQAQRRDX3YWR7LUC7JQ, seg_pos={0 0}, upload=442109]: Uploading piece... Jul 16 10:00:37.312772479 (552h50m12.083413814s) [upload=442109]: Next returned nil segment Jul 16 10:00:37.312790493 (552h50m12.083431828s) [upload=442109]: Waiting for error group managing segments... Jul 16 10:00:37.621481541 (552h50m12.392122876s) [noise=true , address=45.91.116.14:4189 , piece_id=2EOPVJYRHRLZE57BPRX6AO7GAFSNOS5GDJ3ARZAICQUNTAMF3JOA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.622538452 (552h50m12.393179787s) [noise=true , address=45.91.116.8:4047 , piece_id=PZOXUWLARQPBNSN67OX2ANYWWWOUXZ4BJTQAQRRDX3YWR7LUC7JQ, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.625336787 (552h50m12.395978122s) [noise=true , address=45.91.116.11:4118 , piece_id=6AJYPKWRE27Y5VYRHV643PH3MWLR7DQ3CJG7CRNGFUU5DXIPP6LA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.625486202 (552h50m12.396127537s) [noise=true , address=45.91.116.13:4154 , piece_id=7T7C7GQK6MMN2WB5K5GMT7MVIS5LDE7ZVJ3NJXUYSBROTVWED5QA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.643097378 (552h50m12.413738723s) [noise=true , address=45.91.116.12:4227 , piece_id=BUY5Z3M7E6QDKLHNIODRQK3RKJ32XPNXZG7EDKQF6IWFBRAPC67Q, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.646933618 (552h50m12.417574953s) [noise=true , address=77.90.31.11:4553 , piece_id=ER33GQYXW5WQCSR7MTAFCS5LREVU4IEQFZVU25MIK27TOVZJSPMQ, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.656912601 (552h50m12.427553936s) [noise=true , address=77.90.31.14:4661 , piece_id=RQUAB4AGZSNROLUFJEVE2F4YO67LJDYQKWQ4KXDZDUACMAZ36ZXA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.657051858 (552h50m12.427693193s) [noise=true , address=77.90.31.16:4735 , piece_id=HD3S7QZXO4RMPYCL23T3Z74YTJVG7C5MYBWKB244J7WKDNNXCCTA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.658382904 (552h50m12.429024239s) [noise=true , address=77.90.31.18:4806 , piece_id=XUQS52FOQ3QI3V3VXZ2T3OJIML4TFADNTK4IUR4QZMHXOFOGXI7A, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.659112881 (552h50m12.429754216s) [noise=true , address=77.90.31.12:4589 , piece_id=PP4Y5NRJ6JVK4LE6BGTRDN4MCD5JBUY7REZ4BBVQ65YUVZ7UCHWA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.662495022 (552h50m12.433136357s) [noise=true , address=77.90.31.10:4519 , piece_id=Y7FCGTWSZUYYQVVN4A2K5Y6HXYOYUBTDZ4BB3GKZ3CG36M4JG72Q, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.663463916 (552h50m12.434105251s) [noise=true , address=77.90.31.13:4628 , piece_id=JFF7FVEJF4PI3EDCTMGCRWKRT536IKPR367EJDSU6NL5K432I2MQ, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.695800476 (552h50m12.466441811s) [noise=true , address=45.91.116.10:4082 , piece_id=5SQCN7SRWSQWDEUD6CWGKXPIRJZ3AUTMRE3TWOSIE5LB6AHXI56Q, seg_pos={0 0}, upload=442109]: Done uploading piece. err: <nil> Jul 16 10:00:37.695821076 (552h50m12.466462411s) [seg_pos={0 0}, upload=442109]: Segment reached optimal threshold of 13 pieces. Jul 16 10:00:37.695931788 (552h50m12.466573133s) [noise=true , address=104.36.82.9:4005 , piece_id=VRB6S4DFGLIOXJUK3HODA7F3MNEGWCJKLEQ75TUFEPZW334UITVQ, seg_pos={0 0}, upload=442109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: expected piece hash; context canceled; EOF Jul 16 10:00:37.695964340 (552h50m12.466605675s) [noise=true , address=45.91.116.7:4011 , piece_id=FO6J6XC73ANPA23IMGP3D5K5MJVNTX7EBJT3ASYUWAJTSK3PLS2A, seg_pos={0 0}, upload=442109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Jul 16 10:00:37.696027892 (552h50m12.466669227s) [noise=true , address=104.36.82.3:4767 , piece_id=CCBSXDZKI6GJNPKY5N46KQOMOJQU6YGFH3NTRYKIGZJOCGV3FEBA, seg_pos={0 0}, upload=442109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Jul 16 10:00:37.696084500 (552h50m12.466725835s) [seg_pos={0 0}, upload=442109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 16 10:00:37.696142571 (552h50m12.466783896s) [upload=442109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 16 10:00:37.696146709 (552h50m12.466788044s) [upload=442109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 10:00:38.117285014 (552h50m12.887926599s) [upload=442109]: Done uploading segments... Jul 16 10:00:38.117304631 (552h50m12.887945967s) [upload=442109]: Upload finished. err: <nil> Duration: 30.000493227s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.29975083s FAIL (2025-07-16 10:01:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.299786137s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.597888856s FAIL (2025-07-16 10:01:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.597915147s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.826602141s FAIL (2025-07-16 10:00:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.826629794s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.294684397s FAIL (2025-07-16 10:00:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.294707721s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.762484656s FAIL (2025-07-16 10:00:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.762509745s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.889761175s FAIL (2025-07-16 09:59:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.889782827s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 6.43821574s FAIL (2025-07-16 09:59:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.438244465s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 6.12541972s FAIL (2025-07-16 09:59:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.125439218s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.765021516s FAIL (2025-07-16 09:58:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.765057586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.74066375s FAIL (2025-07-16 09:58:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.740739245s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 6.307355169s FAIL (2025-07-16 09:58:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.307384043s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 6.30740567s FAIL (2025-07-16 09:57:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.307435017s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.733434685s FAIL (2025-07-16 09:57:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.733456096s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 6.467043327s FAIL (2025-07-16 09:56:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.467069886s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.721257276s FAIL (2025-07-16 09:56:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.721280619s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration: 5.88862217s FAIL (2025-07-16 09:55:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.888645213s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.73733185s FAIL (2025-07-16 09:55:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.737360975s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 6.261491237s FAIL (2025-07-16 09:54:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.261511646s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.064058203s FAIL (2025-07-16 09:54:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 6.064080226s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 6.320987213s FAIL (2025-07-16 09:53:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.321004565s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........X..................................................] duration: 5.675077222s FAIL (2025-07-16 09:53:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.675105416s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 6.286210227s FAIL (2025-07-16 09:52:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.286234534s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 6.04691226s FAIL (2025-07-16 09:52:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.046929133s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X........X.................................................] duration: 5.76733623s FAIL (2025-07-16 09:51:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.767370156s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 5.946684258s FAIL (2025-07-16 09:51:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.94671113s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........X................................................] duration: 5.722297335s FAIL (2025-07-16 09:51:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.722324596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X........X...............................................] duration: 5.761091379s FAIL (2025-07-16 09:50:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.761118632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X........X..............................................] duration: 5.717994406s FAIL (2025-07-16 09:50:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.718022461s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 6.14476571s FAIL (2025-07-16 09:50:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.144790799s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 6.47386123s FAIL (2025-07-16 09:50:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.473883012s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X........X.............................................] duration: 5.681879515s FAIL (2025-07-16 09:49:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.681899915s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 6.320166195s FAIL (2025-07-16 09:49:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.320190591s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X........X............................................] duration: 5.766600114s FAIL (2025-07-16 09:48:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.766627968s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 6.093397229s FAIL (2025-07-16 09:48:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.093587773s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X........X...........................................] duration: 5.721873706s FAIL (2025-07-16 09:47:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.721906459s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 5.546742645s FAIL (2025-07-16 09:47:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.546768443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X........X..........................................] duration: 5.811689725s FAIL (2025-07-16 09:47:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.811706868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X........X.........................................] duration: 5.712619846s FAIL (2025-07-16 09:46:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.712651006s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 5.810181529s FAIL (2025-07-16 09:46:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.810201799s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X........X........................................] duration: 5.762221464s FAIL (2025-07-16 09:45:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.762247826s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 6.201332249s FAIL (2025-07-16 09:45:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.20135384s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X........X.......................................] duration: 5.657158439s FAIL (2025-07-16 09:44:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.657187454s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 6.483929116s FAIL (2025-07-16 09:44:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.483953853s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X........X......................................] duration: 5.767896093s FAIL (2025-07-16 09:43:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.767921942s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X........X.....................................] duration: 5.672880109s FAIL (2025-07-16 09:43:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.672906189s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 6.343684214s FAIL (2025-07-16 09:42:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.343705295s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X........X....................................] duration: 5.706925531s FAIL (2025-07-16 09:42:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== [upload=441499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 16 09:14:11.682971031 (552h3m46.453612367s) [upload=441499]: Done uploading segments... Jul 16 09:14:11.682988174 (552h3m46.453629509s) [upload=441499]: Upload finished. err: <nil> Jul 16 09:14:26.327645611 (552h4m1.098286946s) [upload=441503]: Starting upload Jul 16 09:14:26.327745291 (552h4m1.098386626s) [upload=441503]: Done starting upload Jul 16 09:14:26.327841095 (552h4m1.09848243s) [upload=441503]: Uploading segments... Jul 16 09:14:26.328174823 (552h4m1.098816178s) [upload=441503]: Got next segment. Inline: false Jul 16 09:14:26.328215471 (552h4m1.098856806s) [upload=441503]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 09:14:26.328220200 (552h4m1.098861535s) [upload=441503]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 09:14:26.564728127 (552h4m1.335369462s) [seg_pos={0 0}, upload=441503]: Begin upload segment... Jul 16 09:14:26.564766972 (552h4m1.335408307s) [seg_pos={0 0}, upload=441503]: Done begin upload segment. Jul 16 09:14:26.564794715 (552h4m1.33543605s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.564803291 (552h4m1.335444626s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565020106 (552h4m1.335661441s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565136919 (552h4m1.335778254s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565295302 (552h4m1.335936637s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565394593 (552h4m1.336035928s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565541323 (552h4m1.336182658s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565650973 (552h4m1.336292308s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565769309 (552h4m1.336410644s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.565947050 (552h4m1.336588385s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566007525 (552h4m1.33664886s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566204452 (552h4m1.336845777s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566224230 (552h4m1.336865555s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566460381 (552h4m1.337101716s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566660565 (552h4m1.3373019s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:26.566854426 (552h4m1.337495761s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Uploading piece... Jul 16 09:14:28.257214459 (552h4m3.027855784s) [upload=441503]: Next returned nil segment Jul 16 09:14:28.257230961 (552h4m3.027872296s) [upload=441503]: Waiting for error group managing segments... Jul 16 09:14:28.607934261 (552h4m3.378575596s) [noise=true , address=45.91.116.10:4082 , piece_id=DSZSA7W3UKX6CWY6OGQSRTWRZY6RHFWXRMRQPTOFLQT6DTBDAD7Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.629612098 (552h4m3.400253433s) [noise=true , address=45.91.116.5:4297 , piece_id=7OANNLUSXNO5VM46DNUMVBKVBPQBO534EQXGRCKUAK5KPA4GWWIA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.631792701 (552h4m3.402434036s) [noise=true , address=45.91.116.9:4263 , piece_id=BJXA6KUUH4O46OPCE44QM7YMT7UWDFQSHZ57WIMGJW5ODB3YZ5YQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.632101531 (552h4m3.402742867s) [noise=true , address=45.91.116.11:4117 , piece_id=WDTV22X7MQOOZRFRBKVXWHSEJIL6UOTEI7DKHZG5CCA37JGRRIEQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696100141 (552h4m3.466741477s) [noise=true , address=45.91.116.4:4938 , piece_id=O6KYEQU2MS3HSHCYTX523PWESXWXIQMLMV26MW2UFSGFLMXGZIOA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.696305525 (552h4m3.46694685s) [noise=true , address=45.91.116.8:4045 , piece_id=B3JV4TBSUAPH4ML6EB7XPZKGKUVUEAMOMMCQOPGAP7A372FUSWDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.707107683 (552h4m3.477749018s) [noise=true , address=104.36.82.11:4113 , piece_id=MOV7NXAYT5QRM7E2R6ZDRPEHQ4TPOSVS3ST4Q2P5ERQSKQKQKYIQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.708545894 (552h4m3.479187219s) [noise=true , address=45.91.116.6:4975 , piece_id=FUUBJYUMOYNEAO3NCFQLK5YB6IINFRUKGYJF5KWWQODZ5ZMEND5A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.711016491 (552h4m3.481657816s) [noise=true , address=104.36.82.3:4767 , piece_id=XHOFSJNPVJOW5FB27UAMJNXVFCIDO342QUXUUPPL7BA2SO7M5SGA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:28.774412499 (552h4m3.545053834s) [noise=true , address=45.91.116.13:4153 , piece_id=V2I2K74WVNEYDVCJQZMMMVY4SX6CXKSMDQ473ZGHWKC3OAEG5JDQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:42.485153717 (552h4m17.255795052s) [noise=true , address=77.90.31.15:4697 , piece_id=CLSBZ7KD5FLCENIJBC3ME7XXBGXDLU6DOCNT6AAYHO6HLPHVDLYQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:54.799737178 (552h4m29.570378513s) [noise=true , address=77.90.31.13:4626 , piece_id=UP7GSEGMCLWR46BYQEGLYZAW7N4F43ZDOKLN6KJAKAH4OMN6AKEA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: <nil> Jul 16 09:14:55.680653470 (552h4m30.451294805s) [noise=true , address=77.90.31.17:4772 , piece_id=LRVXWHVXOFDU6BRZMEFVNLMJPXWKQD3G5LE35BA5UYEH44SRBPZQ, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57790->77.90.31.17:4772: use of closed network connection Jul 16 09:14:55.680714697 (552h4m30.451356032s) [noise=true , address=77.90.31.18:4806 , piece_id=F5XFYNXII3AL6O74Y52JRDIBV44KIC4TBDJRACKDXDORRQQWCJ6A, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60932->77.90.31.18:4806: use of closed network connection Jul 16 09:14:55.680756227 (552h4m30.451397562s) [noise=true , address=77.90.31.21:4878 , piece_id=W22IHNRS57EZ5KMPR5LP5ELE4KDXQUMOYA3W2GNMCIFDIRDS64TA, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58112->77.90.31.21:4878: use of closed network connection Jul 16 09:14:55.680828745 (552h4m30.45147008s) [noise=true , address=77.90.31.10:4518 , piece_id=WTVEFQBQKDZP7NF4DOIKBWKMXIZSBGKUN57IDIPZ5EHDKE2T7E2Q, seg_pos={0 0}, upload=441503]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42046->77.90.31.10:4518: use of closed network connection Jul 16 09:14:55.680871197 (552h4m30.451512532s) [seg_pos={0 0}, upload=441503]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 09:14:55.927916416 (552h4m30.698557751s) [upload=441503]: Done uploading segments... Jul 16 09:14:55.927940001 (552h4m30.698581336s) [upload=441503]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration:* 5.706954256s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 6.315257767s FAIL (2025-07-16 09:41:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.315279179s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)