Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XXX.X.X..............] duration: 5.754783283s FAIL (2025-07-09 00:50:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.75481296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX....................] duration: 30.346071869s FAIL (2025-07-09 00:50:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.346083391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X....X......X...............X...X..X.X.X..X.X...XX....X....] duration: 6.371877136s FAIL (2025-07-09 00:49:37 UTC) orbiter warning: context deadline exceeded Duration: 6.371901092s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX...................] duration: 45.279288271s FAIL (2025-07-09 00:49:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.279305174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................XXX.X.X.............] duration: 5.807819181s FAIL (2025-07-09 00:49:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.80783817s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X....X......X...............X...X..X.X.X..X.X...XX....X...] duration: 6.072784031s FAIL (2025-07-09 00:49:09 UTC) orbiter warning: context deadline exceeded Duration: 6.072807275s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................XXX.X.X............] duration: 5.831033117s FAIL (2025-07-09 00:49:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.83107089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX..................] duration: 39.29241898s FAIL (2025-07-09 00:48:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.292439008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................XXX.X.X...........] duration: 7.168262259s FAIL (2025-07-09 00:48:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.168281115s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..X....X......X...............X...X..X.X.X..X.X...XX....X..] duration: 5.937531795s FAIL (2025-07-09 00:48:11 UTC) orbiter warning: context deadline exceeded Duration: 5.937552034s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................XXX.X.X..........] duration: 5.899639601s FAIL (2025-07-09 00:48:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.899656963s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X..X....X......X...............X...X..X.X.X..X.X...XX....X.] duration: 28.426954871s FAIL (2025-07-09 00:48:00 UTC) orbiter warning: context deadline exceeded Duration: 28.42697571s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................XXX.X.X.........] duration: 18.953597s FAIL (2025-07-09 00:47:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 18.9536235s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX.................] duration: 49.447672847s FAIL (2025-07-09 00:47:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.447686333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXX.X.X........] duration: 14.511820619s FAIL (2025-07-09 00:47:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.511845737s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X..X....X......X...............X...X..X.X.X..X.X...XX....X] duration: 30.000693207s FAIL (2025-07-09 00:47:26 UTC) orbiter warning: context deadline exceeded Duration: 30.00073098s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX................] duration: 39.447894656s FAIL (2025-07-09 00:46:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.447909976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................XXX.X.X.......] duration: 25.507005673s FAIL (2025-07-09 00:46:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 25.507026133s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX...............] duration: 46.550894474s FAIL (2025-07-09 00:46:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.55090781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.X..X....X......X...............X...X..X.X.X..X.X...XX....] duration: 20.492980076s FAIL (2025-07-09 00:46:06 UTC) orbiter warning: context deadline exceeded Duration: 20.492999804s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX..............] duration: 44.12017257s FAIL (2025-07-09 00:45:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.120193341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXX.X.X......] duration: 14.170759986s FAIL (2025-07-09 00:45:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.170818527s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX.X..X....X......X...............X...X..X.X.X..X.X...XX...] duration: 13.126422334s FAIL (2025-07-09 00:45:06 UTC) orbiter warning: context deadline exceeded Duration: 13.126437412s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX.............] duration: 28.750223063s FAIL (2025-07-09 00:44:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.7502474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXX.X..X....X......X...............X...X..X.X.X..X.X...XX..] duration: 24.53336224s FAIL (2025-07-09 00:44:37 UTC) orbiter warning: context deadline exceeded Duration: 24.533381467s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................XXX.X.X.....] duration: 26.335687988s FAIL (2025-07-09 00:44:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 26.33572029s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX............] duration: 28.550118806s FAIL (2025-07-09 00:44:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.550130108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XXXX.X..X....X......X...............X...X..X.X.X..X.X...XX.] duration: 28.830513459s FAIL (2025-07-09 00:43:57 UTC) orbiter warning: context deadline exceeded Duration: 28.830538186s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [X.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX...........] duration: 35.508962695s FAIL (2025-07-09 00:43:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.508974971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [XX.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX..........] duration: 35.298111539s FAIL (2025-07-09 00:43:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.298122981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.XXXX.X..X....X......X...............X...X..X.X.X..X.X...XX] duration: 30.00037203s FAIL (2025-07-09 00:43:11 UTC) orbiter warning: context deadline exceeded Duration: 30.000395585s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XXX.X.X....] duration: 7.198698431s FAIL (2025-07-09 00:43:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.198723228s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [XXX.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX.........] duration: 38.29352403s FAIL (2025-07-09 00:42:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.293546764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XXX.X.X...] duration: 5.749972091s FAIL (2025-07-09 00:42:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.74998823s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XXX.X.X..] duration: 21.423752592s FAIL (2025-07-09 00:42:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 21.423781708s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [.XXX.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX........] duration: 42.094245218s FAIL (2025-07-09 00:41:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.094263603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.XXXX.X..X....X......X...............X...X..X.X.X..X.X...X] duration: 30.001221305s FAIL (2025-07-09 00:41:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001238548s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXX.X.X.] duration: 6.060251783s FAIL (2025-07-09 00:41:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.060278604s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [X.XXX.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX.......] duration: 36.737212706s FAIL (2025-07-09 00:41:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.737233996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX.XXXX.X..X....X......X...............X...X..X.X.X..X.X...] duration: 26.677642402s FAIL (2025-07-09 00:41:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.677665075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXX.X.X] duration: 30.235283178s FAIL (2025-07-09 00:40:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 39:26.363507162 (375h29m1.134148507s) [upload=300391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:39:26.363513754 (375h29m1.134155089s) [upload=300391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:39:26.664388326 (375h29m1.435029661s) [upload=300391]: Done uploading segments... Jul 9 00:39:26.664407853 (375h29m1.435049188s) [upload=300391]: Upload finished. err: <nil> Jul 9 00:40:22.831115797 (375h29m57.601757142s) [upload=300409]: Starting upload Jul 9 00:40:22.831211139 (375h29m57.601852484s) [upload=300409]: Done starting upload Jul 9 00:40:22.831297705 (375h29m57.60193906s) [upload=300409]: Uploading segments... Jul 9 00:40:22.831515621 (375h29m57.602156976s) [upload=300409]: Got next segment. Inline: false Jul 9 00:40:22.831567641 (375h29m57.602208996s) [upload=300409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:40:22.831574394 (375h29m57.60221573s) [upload=300409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:40:23.136866206 (375h29m57.907507541s) [seg_pos={0 0}, upload=300409]: Begin upload segment... Jul 9 00:40:23.136907616 (375h29m57.907548951s) [seg_pos={0 0}, upload=300409]: Done begin upload segment. Jul 9 00:40:23.136933185 (375h29m57.90757452s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.136949325 (375h29m57.90759066s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137300748 (375h29m57.907942083s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137357226 (375h29m57.907998551s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137573079 (375h29m57.908214414s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137669523 (375h29m57.908310858s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137935562 (375h29m57.908576898s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.137991359 (375h29m57.908632694s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138195048 (375h29m57.908836384s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138296653 (375h29m57.908937988s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138457411 (375h29m57.909098746s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138578562 (375h29m57.909219887s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138658365 (375h29m57.9092997s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138822549 (375h29m57.909463884s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.138937289 (375h29m57.909578614s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:23.139137432 (375h29m57.909778767s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Uploading piece... Jul 9 00:40:24.925431157 (375h29m59.696072493s) [upload=300409]: Next returned nil segment Jul 9 00:40:24.925452939 (375h29m59.696094284s) [upload=300409]: Waiting for error group managing segments... Jul 9 00:40:25.211762407 (375h29m59.982403742s) [noise=true , address=45.91.116.11:4118 , piece_id=NSXCFVVUPVXPC75SQ2HZEFDSG7WYOOOGVWHSTE2GQLJ4AMW2DR7A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219531349 (375h29m59.990172684s) [noise=true , address=45.91.116.14:4190 , piece_id=3SNJWXBD6IM37H2OB3FMQPJNRXROQMKLCLREJNOXR4EVGFZ3EWHQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.219825963 (375h29m59.990467298s) [noise=true , address=45.91.116.10:4083 , piece_id=QB7IZNTRXUZUB2I7F3ENTX5VUMR2SQ4ZMXYX63OZ7BBVBF6OQNEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.223483851 (375h29m59.994125176s) [noise=true , address=45.91.116.4:4937 , piece_id=K32VK6T6D42USHFPN2ZKVNASBDVGFZIPY7ANGLDBUVXOPKTBNFYA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.233603090 (375h30m0.004244425s) [noise=true , address=45.91.116.10:4082 , piece_id=U4FCLI2EKP4BLSRVQIM4NDGP2RFTXCBMT52MES6AWXRFHNHBG3UA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.239221890 (375h30m0.009863225s) [noise=true , address=45.91.116.5:4299 , piece_id=R7W27CAHTJ2J4A7R5ED4MCKYTJFIK7ATKYSOFQQQ4UA7SKTR2L7Q, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.283055430 (375h30m0.053696765s) [noise=true , address=77.90.31.18:4805 , piece_id=MT357YKF45EP7DJGF6IVU5CXL34JKSAOJKVGDC6K4J2A3TL7GCPA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305071960 (375h30m0.075713295s) [noise=true , address=45.91.116.8:4045 , piece_id=BL5CCIPZ6CKYUUR472N2YTJIVG6YGSAU4C5TCVHZLS6NKMMP6HEQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.305351565 (375h30m0.0759929s) [noise=true , address=45.91.116.7:4010 , piece_id=3CZOPS4457RC53IRBK65O5HPKEKAONDNMVXO6EMLYZDC7VCQIXBQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:25.310763678 (375h30m0.081405013s) [noise=true , address=45.91.116.13:4153 , piece_id=QS25MZIB75FXI5NSQX727J4LO2O3TXLWOERNBFIZRGGXOKUJN4BA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:27.129473449 (375h30m1.900114784s) [noise=true , address=104.36.82.16:4294 , piece_id=QGZLYRKLPR6US4HR5LBYUPD2OFSKT4VE6WGOKLKAPCFWHC3QH2LA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:42.274686523 (375h30m17.045327858s) [noise=true , address=104.36.82.7:4910 , piece_id=E45U3F5CXZRUFHLKP67643QREDMFI54ZA4KHK4HT4VF5BXHTZTIA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: <nil> Jul 9 00:40:52.188063012 (375h30m26.958704347s) [noise=true , address=104.36.82.14:4222 , piece_id=EZ65N3KNF7L4K7Y2SJIESME7I7ITAWS4IVMN2CSJS2MA26UENHKQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 00:40:52.188129919 (375h30m26.958771244s) [noise=true , address=104.36.82.2:4729 , piece_id=S4MWPIVNLQ3QVPTL3WZ7ASCBYBP7W3FJXWFANS52MNWLZW75CU3A, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->104.36.82.2:4729: use of closed network connection Jul 9 00:40:52.188272542 (375h30m26.958913877s) [noise=true , address=104.36.82.14:4223 , piece_id=WVEVHZDJWVX6BNXU5JANS2EONIGDELN4A6O27EEAZM7HVAGIUTVQ, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57838->104.36.82.14:4223: use of closed network connection Jul 9 00:40:52.188319602 (375h30m26.958960927s) [noise=true , address=104.36.82.9:4006 , piece_id=PTCVRETYYRJVDYD3P5GJTFTJP5FES5NVOLZOEXFP3JXJLE3QKEXA, seg_pos={0 0}, upload=300409]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49218->104.36.82.9:4006: use of closed network connection Jul 9 00:40:52.188364118 (375h30m26.959005443s) [seg_pos={0 0}, upload=300409]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:40:52.422425287 (375h30m27.193066622s) [upload=300409]: Done uploading segments... Jul 9 00:40:52.422450515 (375h30m27.19309184s) [upload=300409]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235307405s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX.XXXX.X..X....X......X...............X...X..X.X.X..X.X..] duration: 19.191940523s FAIL (2025-07-09 00:40:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.191964547s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [XX.XXX.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX......] duration: 33.091384941s FAIL (2025-07-09 00:40:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.091402455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX.X.] duration: 16.267563123s FAIL (2025-07-09 00:39:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ng segments... Jul 9 00:38:10.357742952 (375h27m45.128384287s) [upload=300377]: Got next segment. Inline: false Jul 9 00:38:10.357765315 (375h27m45.12840664s) [upload=300377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:38:10.357769443 (375h27m45.128410768s) [upload=300377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:38:10.601812176 (375h27m45.372453512s) [seg_pos={0 0}, upload=300377]: Begin upload segment... Jul 9 00:38:10.601858105 (375h27m45.37249944s) [seg_pos={0 0}, upload=300377]: Done begin upload segment. Jul 9 00:38:10.601884535 (375h27m45.37252587s) [noise=true , address=104.36.82.12:4151 , piece_id=EL5BDCNOYCZUK5T5X7DDF2OZCRUPXEX2FVR7L6TUQC5KRAHC2VDA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.601890446 (375h27m45.372531781s) [noise=true , address=104.36.82.15:4258 , piece_id=425O4HGLNGILDFKV47YOM7KX7XRZKVUOLCMXHZ2DDGQZW32MNXUQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.601942246 (375h27m45.372583581s) [noise=true , address=45.91.116.12:4226 , piece_id=6OL4SWHBP345IK53ZRJ52BD3L222CGYEGEJKZ5P4AD6T7CX4YKOQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602176624 (375h27m45.372817959s) [noise=true , address=45.91.116.9:4263 , piece_id=KAHNUQXSFZRKG7NMJI4BAY6ZNKTNKCNTVH6DGF3YQXYEOYMJ5JFQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602179950 (375h27m45.372821285s) [noise=true , address=77.90.31.16:4733 , piece_id=PVL2HBMNQQGWK4I6FYTLFXFYY25WNJABEKBEYD7AMMWJWDEW6Q6Q, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602365906 (375h27m45.373007241s) [noise=true , address=104.36.82.7:4909 , piece_id=MXCKG5QCWL5DFO2XOFISZUN2F7QFEFBTVXY3KVHKNU77DSG7B65A, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602369873 (375h27m45.373011208s) [noise=true , address=77.90.31.10:4519 , piece_id=6HJ5TQ3UGPXGQJOEXCBSCPIJGBKAZQJEXVVXZFGDA5V64TPNMAXA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602561049 (375h27m45.373202374s) [noise=true , address=45.91.116.7:4011 , piece_id=U47FXN3KHT3D4QQ77CHHHIXPKYJ7LHUDB4RW47IQFKNHUMSNFEHA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602567202 (375h27m45.373208527s) [noise=true , address=45.91.116.4:4937 , piece_id=MPMEVKMO5WMNQFHEDS2IOJ5XVNYEDQGSRKY6UPABDHVTVDTITN6Q, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602757294 (375h27m45.373398629s) [noise=true , address=45.91.116.8:4045 , piece_id=5OOWYVN3GCWJAX2IAPTAR7P57YP3WUPJZ63TNYBDT6QHRANTKPOA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602945555 (375h27m45.37358689s) [noise=true , address=104.36.82.14:4221 , piece_id=QFZWQRICFITWSFXORFQCFWHJAVHQWC6BGPBF5W57WNYGSDEO5GCQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.603133394 (375h27m45.373774729s) [noise=true , address=45.91.116.11:4117 , piece_id=5VGNXT6A227OUGO3ZK3IZMLJO3QRDCV53L3EPQROMTW6VYVXLAIQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.603329549 (375h27m45.373970885s) [noise=true , address=104.36.82.15:4257 , piece_id=WSEC4OFGEXICY3HH4YJ7REMBY7M3252GDOYSCNZG65TMNSYO3EGQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.603517549 (375h27m45.374158884s) [noise=true , address=45.91.116.14:4191 , piece_id=RUTFACKHMLUCVND6WCMFNWW7F7SZFUQD2QJAKOJWWEZYCHG2BU4A, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.605916178 (375h27m45.376557513s) [noise=true , address=45.91.116.6:4975 , piece_id=NA5RZ6CVD7JMJY2W4UJEEC4ZEH53S7WBJMIYE7HUMYJ2FZPMHQKQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.607647451 (375h27m45.378288786s) [noise=true , address=45.91.116.5:4298 , piece_id=ULWY2PCOMIJHY5YRJ4STUMLEYF2RBTIO3BMX24JCIQUCVNGYD5YA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:12.356172531 (375h27m47.126813866s) [upload=300377]: Next returned nil segment Jul 9 00:38:12.356191006 (375h27m47.126832331s) [upload=300377]: Waiting for error group managing segments... Jul 9 00:38:12.691725082 (375h27m47.462366417s) [noise=true , address=45.91.116.4:4937 , piece_id=MPMEVKMO5WMNQFHEDS2IOJ5XVNYEDQGSRKY6UPABDHVTVDTITN6Q, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.696569470 (375h27m47.467210805s) [noise=true , address=45.91.116.11:4117 , piece_id=5VGNXT6A227OUGO3ZK3IZMLJO3QRDCV53L3EPQROMTW6VYVXLAIQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.698488622 (375h27m47.469129957s) [noise=true , address=45.91.116.14:4191 , piece_id=RUTFACKHMLUCVND6WCMFNWW7F7SZFUQD2QJAKOJWWEZYCHG2BU4A, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.700364510 (375h27m47.471005846s) [noise=true , address=45.91.116.7:4011 , piece_id=U47FXN3KHT3D4QQ77CHHHIXPKYJ7LHUDB4RW47IQFKNHUMSNFEHA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.701659719 (375h27m47.472301055s) [noise=true , address=45.91.116.9:4263 , piece_id=KAHNUQXSFZRKG7NMJI4BAY6ZNKTNKCNTVH6DGF3YQXYEOYMJ5JFQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.750358540 (375h27m47.520999865s) [noise=true , address=77.90.31.16:4733 , piece_id=PVL2HBMNQQGWK4I6FYTLFXFYY25WNJABEKBEYD7AMMWJWDEW6Q6Q, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.758209701 (375h27m47.528851036s) [noise=true , address=77.90.31.10:4519 , piece_id=6HJ5TQ3UGPXGQJOEXCBSCPIJGBKAZQJEXVVXZFGDA5V64TPNMAXA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.761448086 (375h27m47.532089421s) [noise=true , address=45.91.116.8:4045 , piece_id=5OOWYVN3GCWJAX2IAPTAR7P57YP3WUPJZ63TNYBDT6QHRANTKPOA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.771872530 (375h27m47.542513865s) [noise=true , address=45.91.116.6:4975 , piece_id=NA5RZ6CVD7JMJY2W4UJEEC4ZEH53S7WBJMIYE7HUMYJ2FZPMHQKQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.773563145 (375h27m47.54420448s) [noise=true , address=45.91.116.12:4226 , piece_id=6OL4SWHBP345IK53ZRJ52BD3L222CGYEGEJKZ5P4AD6T7CX4YKOQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.774132985 (375h27m47.54477432s) [noise=true , address=45.91.116.5:4298 , piece_id=ULWY2PCOMIJHY5YRJ4STUMLEYF2RBTIO3BMX24JCIQUCVNGYD5YA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:39.704937009 (375h28m14.475578344s) [noise=true , address=104.36.82.12:4151 , piece_id=EL5BDCNOYCZUK5T5X7DDF2OZCRUPXEX2FVR7L6TUQC5KRAHC2VDA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:52996->104.36.82.12:4151: use of closed network connection; write tcp 188.166.225.170:52996->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52996->104.36.82.12:4151: use of closed network connection Jul 9 00:38:39.704983659 (375h28m14.475624984s) [noise=true , address=104.36.82.7:4909 , piece_id=MXCKG5QCWL5DFO2XOFISZUN2F7QFEFBTVXY3KVHKNU77DSG7B65A, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:42158->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:42158->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42158->104.36.82.7:4909: use of closed network connection Jul 9 00:38:39.705028755 (375h28m14.47567009s) [noise=true , address=104.36.82.15:4257 , piece_id=WSEC4OFGEXICY3HH4YJ7REMBY7M3252GDOYSCNZG65TMNSYO3EGQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:57116->104.36.82.15:4257: use of closed network connection; write tcp 188.166.225.170:57116->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57116->104.36.82.15:4257: use of closed network connection Jul 9 00:38:39.705073600 (375h28m14.475714935s) [noise=true , address=104.36.82.15:4258 , piece_id=425O4HGLNGILDFKV47YOM7KX7XRZKVUOLCMXHZ2DDGQZW32MNXUQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:48076->104.36.82.15:4258: use of closed network connection; write tcp 188.166.225.170:48076->104.36.82.15:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48076->104.36.82.15:4258: use of closed network connection Jul 9 00:38:39.705100482 (375h28m14.475741827s) [noise=true , address=104.36.82.14:4221 , piece_id=QFZWQRICFITWSFXORFQCFWHJAVHQWC6BGPBF5W57WNYGSDEO5GCQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:60148->104.36.82.14:4221: use of closed network connection; write tcp 188.166.225.170:60148->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60148->104.36.82.14:4221: use of closed network connection Jul 9 00:38:39.705135348 (375h28m14.475776693s) [seg_pos={0 0}, upload=300377]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 00:38:39.941063649 (375h28m14.711704984s) [upload=300377]: Done uploading segments... Jul 9 00:38:39.941089628 (375h28m14.711730953s) [upload=300377]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 16.267587921s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...XX.XXXX.X..X....X......X...............X...X..X.X.X..X.X.] duration: 23.552750428s FAIL (2025-07-09 00:39:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.552773724s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [XXX.XXX.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX.....] duration: 18.085882793s FAIL (2025-07-09 00:38:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.085905818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXX.X] duration: 30.237436994s FAIL (2025-07-09 00:38:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ng segments... Jul 9 00:38:10.357742952 (375h27m45.128384287s) [upload=300377]: Got next segment. Inline: false Jul 9 00:38:10.357765315 (375h27m45.12840664s) [upload=300377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:38:10.357769443 (375h27m45.128410768s) [upload=300377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:38:10.601812176 (375h27m45.372453512s) [seg_pos={0 0}, upload=300377]: Begin upload segment... Jul 9 00:38:10.601858105 (375h27m45.37249944s) [seg_pos={0 0}, upload=300377]: Done begin upload segment. Jul 9 00:38:10.601884535 (375h27m45.37252587s) [noise=true , address=104.36.82.12:4151 , piece_id=EL5BDCNOYCZUK5T5X7DDF2OZCRUPXEX2FVR7L6TUQC5KRAHC2VDA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.601890446 (375h27m45.372531781s) [noise=true , address=104.36.82.15:4258 , piece_id=425O4HGLNGILDFKV47YOM7KX7XRZKVUOLCMXHZ2DDGQZW32MNXUQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.601942246 (375h27m45.372583581s) [noise=true , address=45.91.116.12:4226 , piece_id=6OL4SWHBP345IK53ZRJ52BD3L222CGYEGEJKZ5P4AD6T7CX4YKOQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602176624 (375h27m45.372817959s) [noise=true , address=45.91.116.9:4263 , piece_id=KAHNUQXSFZRKG7NMJI4BAY6ZNKTNKCNTVH6DGF3YQXYEOYMJ5JFQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602179950 (375h27m45.372821285s) [noise=true , address=77.90.31.16:4733 , piece_id=PVL2HBMNQQGWK4I6FYTLFXFYY25WNJABEKBEYD7AMMWJWDEW6Q6Q, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602365906 (375h27m45.373007241s) [noise=true , address=104.36.82.7:4909 , piece_id=MXCKG5QCWL5DFO2XOFISZUN2F7QFEFBTVXY3KVHKNU77DSG7B65A, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602369873 (375h27m45.373011208s) [noise=true , address=77.90.31.10:4519 , piece_id=6HJ5TQ3UGPXGQJOEXCBSCPIJGBKAZQJEXVVXZFGDA5V64TPNMAXA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602561049 (375h27m45.373202374s) [noise=true , address=45.91.116.7:4011 , piece_id=U47FXN3KHT3D4QQ77CHHHIXPKYJ7LHUDB4RW47IQFKNHUMSNFEHA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602567202 (375h27m45.373208527s) [noise=true , address=45.91.116.4:4937 , piece_id=MPMEVKMO5WMNQFHEDS2IOJ5XVNYEDQGSRKY6UPABDHVTVDTITN6Q, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602757294 (375h27m45.373398629s) [noise=true , address=45.91.116.8:4045 , piece_id=5OOWYVN3GCWJAX2IAPTAR7P57YP3WUPJZ63TNYBDT6QHRANTKPOA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.602945555 (375h27m45.37358689s) [noise=true , address=104.36.82.14:4221 , piece_id=QFZWQRICFITWSFXORFQCFWHJAVHQWC6BGPBF5W57WNYGSDEO5GCQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.603133394 (375h27m45.373774729s) [noise=true , address=45.91.116.11:4117 , piece_id=5VGNXT6A227OUGO3ZK3IZMLJO3QRDCV53L3EPQROMTW6VYVXLAIQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.603329549 (375h27m45.373970885s) [noise=true , address=104.36.82.15:4257 , piece_id=WSEC4OFGEXICY3HH4YJ7REMBY7M3252GDOYSCNZG65TMNSYO3EGQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.603517549 (375h27m45.374158884s) [noise=true , address=45.91.116.14:4191 , piece_id=RUTFACKHMLUCVND6WCMFNWW7F7SZFUQD2QJAKOJWWEZYCHG2BU4A, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.605916178 (375h27m45.376557513s) [noise=true , address=45.91.116.6:4975 , piece_id=NA5RZ6CVD7JMJY2W4UJEEC4ZEH53S7WBJMIYE7HUMYJ2FZPMHQKQ, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:10.607647451 (375h27m45.378288786s) [noise=true , address=45.91.116.5:4298 , piece_id=ULWY2PCOMIJHY5YRJ4STUMLEYF2RBTIO3BMX24JCIQUCVNGYD5YA, seg_pos={0 0}, upload=300377]: Uploading piece... Jul 9 00:38:12.356172531 (375h27m47.126813866s) [upload=300377]: Next returned nil segment Jul 9 00:38:12.356191006 (375h27m47.126832331s) [upload=300377]: Waiting for error group managing segments... Jul 9 00:38:12.691725082 (375h27m47.462366417s) [noise=true , address=45.91.116.4:4937 , piece_id=MPMEVKMO5WMNQFHEDS2IOJ5XVNYEDQGSRKY6UPABDHVTVDTITN6Q, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.696569470 (375h27m47.467210805s) [noise=true , address=45.91.116.11:4117 , piece_id=5VGNXT6A227OUGO3ZK3IZMLJO3QRDCV53L3EPQROMTW6VYVXLAIQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.698488622 (375h27m47.469129957s) [noise=true , address=45.91.116.14:4191 , piece_id=RUTFACKHMLUCVND6WCMFNWW7F7SZFUQD2QJAKOJWWEZYCHG2BU4A, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.700364510 (375h27m47.471005846s) [noise=true , address=45.91.116.7:4011 , piece_id=U47FXN3KHT3D4QQ77CHHHIXPKYJ7LHUDB4RW47IQFKNHUMSNFEHA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.701659719 (375h27m47.472301055s) [noise=true , address=45.91.116.9:4263 , piece_id=KAHNUQXSFZRKG7NMJI4BAY6ZNKTNKCNTVH6DGF3YQXYEOYMJ5JFQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.750358540 (375h27m47.520999865s) [noise=true , address=77.90.31.16:4733 , piece_id=PVL2HBMNQQGWK4I6FYTLFXFYY25WNJABEKBEYD7AMMWJWDEW6Q6Q, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.758209701 (375h27m47.528851036s) [noise=true , address=77.90.31.10:4519 , piece_id=6HJ5TQ3UGPXGQJOEXCBSCPIJGBKAZQJEXVVXZFGDA5V64TPNMAXA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.761448086 (375h27m47.532089421s) [noise=true , address=45.91.116.8:4045 , piece_id=5OOWYVN3GCWJAX2IAPTAR7P57YP3WUPJZ63TNYBDT6QHRANTKPOA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.771872530 (375h27m47.542513865s) [noise=true , address=45.91.116.6:4975 , piece_id=NA5RZ6CVD7JMJY2W4UJEEC4ZEH53S7WBJMIYE7HUMYJ2FZPMHQKQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.773563145 (375h27m47.54420448s) [noise=true , address=45.91.116.12:4226 , piece_id=6OL4SWHBP345IK53ZRJ52BD3L222CGYEGEJKZ5P4AD6T7CX4YKOQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:12.774132985 (375h27m47.54477432s) [noise=true , address=45.91.116.5:4298 , piece_id=ULWY2PCOMIJHY5YRJ4STUMLEYF2RBTIO3BMX24JCIQUCVNGYD5YA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: <nil> Jul 9 00:38:39.704937009 (375h28m14.475578344s) [noise=true , address=104.36.82.12:4151 , piece_id=EL5BDCNOYCZUK5T5X7DDF2OZCRUPXEX2FVR7L6TUQC5KRAHC2VDA, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:52996->104.36.82.12:4151: use of closed network connection; write tcp 188.166.225.170:52996->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52996->104.36.82.12:4151: use of closed network connection Jul 9 00:38:39.704983659 (375h28m14.475624984s) [noise=true , address=104.36.82.7:4909 , piece_id=MXCKG5QCWL5DFO2XOFISZUN2F7QFEFBTVXY3KVHKNU77DSG7B65A, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:42158->104.36.82.7:4909: use of closed network connection; write tcp 188.166.225.170:42158->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42158->104.36.82.7:4909: use of closed network connection Jul 9 00:38:39.705028755 (375h28m14.47567009s) [noise=true , address=104.36.82.15:4257 , piece_id=WSEC4OFGEXICY3HH4YJ7REMBY7M3252GDOYSCNZG65TMNSYO3EGQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:57116->104.36.82.15:4257: use of closed network connection; write tcp 188.166.225.170:57116->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57116->104.36.82.15:4257: use of closed network connection Jul 9 00:38:39.705073600 (375h28m14.475714935s) [noise=true , address=104.36.82.15:4258 , piece_id=425O4HGLNGILDFKV47YOM7KX7XRZKVUOLCMXHZ2DDGQZW32MNXUQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:48076->104.36.82.15:4258: use of closed network connection; write tcp 188.166.225.170:48076->104.36.82.15:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48076->104.36.82.15:4258: use of closed network connection Jul 9 00:38:39.705100482 (375h28m14.475741827s) [noise=true , address=104.36.82.14:4221 , piece_id=QFZWQRICFITWSFXORFQCFWHJAVHQWC6BGPBF5W57WNYGSDEO5GCQ, seg_pos={0 0}, upload=300377]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:60148->104.36.82.14:4221: use of closed network connection; write tcp 188.166.225.170:60148->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60148->104.36.82.14:4221: use of closed network connection Jul 9 00:38:39.705135348 (375h28m14.475776693s) [seg_pos={0 0}, upload=300377]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 9 00:38:39.941063649 (375h28m14.711704984s) [upload=300377]: Done uploading segments... Jul 9 00:38:39.941089628 (375h28m14.711730953s) [upload=300377]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.237461372s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XX.XXXX.X..X....X......X...............X...X..X.X.X..X.X] duration: 30.001365394s FAIL (2025-07-09 00:38:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001384981s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 44 out of 60 is failed (threshold=3) [XXXX.XXX.XX.XXXXXXXXXXXXXXXXXXX.X....X..XXXXXXX.XXXX.XXX....] duration: 17.994619877s FAIL (2025-07-09 00:38:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.994637159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 5.765030581s FAIL (2025-07-09 00:37:49 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 (375h26m3.249898238s) [noise=true , address=104.36.82.9:4007 , piece_id=4NUW6U5LI2IIWDZ25V6NBWHACEQEXUZTJJ36TSXB5TEPGMSMHKJQ, seg_pos={0 0}, upload=300349]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:40722->104.36.82.9:4007: use of closed network connection; write tcp 188.166.225.170:40722->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40722->104.36.82.9:4007: use of closed network connection Jul 9 00:36:28.479284506 (375h26m3.249925841s) [seg_pos={0 0}, upload=300349]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 00:36:28.733294570 (375h26m3.503935905s) [upload=300349]: Done uploading segments... Jul 9 00:36:28.733319668 (375h26m3.503961003s) [upload=300349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 9 00:36:29.580601982 (375h26m4.351243317s) [upload=300357]: Starting upload Jul 9 00:36:29.580663510 (375h26m4.351304835s) [upload=300357]: Done starting upload Jul 9 00:36:29.580765425 (375h26m4.35140676s) [upload=300357]: Uploading segments... Jul 9 00:36:29.580886005 (375h26m4.35152734s) [upload=300357]: Got next segment. Inline: false Jul 9 00:36:29.580917857 (375h26m4.351559192s) [upload=300357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 00:36:29.580923197 (375h26m4.351564532s) [upload=300357]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 00:36:29.818509448 (375h26m4.589150783s) [seg_pos={0 0}, upload=300357]: Begin upload segment... Jul 9 00:36:29.818578650 (375h26m4.589219985s) [seg_pos={0 0}, upload=300357]: Done begin upload segment. Jul 9 00:36:29.818603778 (375h26m4.589245103s) [noise=true , address=77.90.31.10:4517 , piece_id=Z77S2VYE3Y6MNBJ3E2LKYGWIMY6IIEVO3LOMNDE3D57SUCGGPRUA, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.818977123 (375h26m4.589618458s) [noise=true , address=77.90.31.21:4879 , piece_id=E2RFU72GA5KHGEHN6SZ2DRDPU3ANYV7TBTVY5TJFEOUNBUFR25LQ, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.819208746 (375h26m4.589850081s) [noise=true , address=45.91.116.11:4117 , piece_id=F2K2VFZR4HJV3J45XB5X46XK7AYG7XQEP5N4JYBW4FAF3SOVRIPQ, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.819437033 (375h26m4.590078368s) [noise=true , address=104.36.82.16:4294 , piece_id=YDM4PBG7FQMNILDJDTPH3EOLXKAZ3FULLFUTYFUOHPCSM5ICGR4Q, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.819696239 (375h26m4.590337574s) [noise=true , address=104.36.82.8:4972 , piece_id=3WE5NH4IDV2YUI6Q2C3FGOUADXIPCTP7DIJ32MSAHASWB4HBQOHQ, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.819940566 (375h26m4.590581901s) [noise=true , address=77.90.31.19:4843 , piece_id=336ULA34A37WNNMZW74TXORG24UTHHDPRBBVEZBW44B3A4MVWUCA, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.820166248 (375h26m4.590807573s) [noise=true , address=45.91.116.4:4938 , piece_id=FK7EJMPXVENEXMOOGOP5H4VQMALQK45CQGSQIRBITP5X5CV3RO4A, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.820570412 (375h26m4.591211747s) [noise=true , address=45.91.116.7:4009 , piece_id=XXGMERGTXCXZ5KHY26ZTXDO5DISVXLCALRBKG6XKOILKKYUQO5EA, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.820948204 (375h26m4.591589539s) [noise=true , address=104.36.82.11:4114 , piece_id=7S6NXTZY7RPMFDYHUBC4A6G5JHOYWJRH455H5PYMU32FIE4KXQ2A, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.821149439 (375h26m4.591790774s) [noise=true , address=45.91.116.8:4047 , piece_id=ADSEVJ2QP3AJ3E5R7OXEJEQPETW6SE5373OP635TGVJQW2DYRBIQ, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.821365412 (375h26m4.592006737s) [noise=true , address=77.90.31.16:4735 , piece_id=JJ2XV7T4VQX53VL3QEEHCPNMKVV6YF67YWNXYJRF5SP2YEAYW22A, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.821552991 (375h26m4.592194326s) [noise=true , address=45.91.116.12:4227 , piece_id=ARF37W6QX73MLI4EXX2W7L6KX5QUP66NZH62WN6TLEEBULOTHEEA, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.821819481 (375h26m4.592460816s) [noise=true , address=104.36.82.8:4970 , piece_id=XRIL73IEWUKUO73EU6457GO7YVIUWB34XY5Y227K26OOAYX7JT6Q, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.822075751 (375h26m4.592717086s) [noise=true , address=45.91.116.10:4083 , piece_id=KYJHYFP3H4RE73TSZINPGCSG4S7OVSHZEDACVZ7JX6HOCMZAK7TQ, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.822287697 (375h26m4.592929032s) [noise=true , address=45.91.116.5:4299 , piece_id=LASSSRSDTFT7RCSWTFZOVDIQ2FS65WWZ2VLP2VAXJZXWBKM3WKZA, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:29.822513639 (375h26m4.593154964s) [noise=true , address=104.36.82.4:4801 , piece_id=V4MBV5ZRQR32IOV2L7NYAQKRJAKTE3DZOKDWZO6DP3KI26N7ES3A, seg_pos={0 0}, upload=300357]: Uploading piece... Jul 9 00:36:31.610466701 (375h26m6.381108026s) [upload=300357]: Next returned nil segment Jul 9 00:36:31.610485658 (375h26m6.381126993s) [upload=300357]: Waiting for error group managing segments... Jul 9 00:36:31.906852107 (375h26m6.677493442s) [noise=true , address=45.91.116.4:4938 , piece_id=FK7EJMPXVENEXMOOGOP5H4VQMALQK45CQGSQIRBITP5X5CV3RO4A, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.908513406 (375h26m6.679154741s) [noise=true , address=45.91.116.5:4299 , piece_id=LASSSRSDTFT7RCSWTFZOVDIQ2FS65WWZ2VLP2VAXJZXWBKM3WKZA, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.923298293 (375h26m6.693939628s) [noise=true , address=45.91.116.8:4047 , piece_id=ADSEVJ2QP3AJ3E5R7OXEJEQPETW6SE5373OP635TGVJQW2DYRBIQ, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.925150827 (375h26m6.695792162s) [noise=true , address=45.91.116.10:4083 , piece_id=KYJHYFP3H4RE73TSZINPGCSG4S7OVSHZEDACVZ7JX6HOCMZAK7TQ, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.930127868 (375h26m6.700769203s) [noise=true , address=45.91.116.12:4227 , piece_id=ARF37W6QX73MLI4EXX2W7L6KX5QUP66NZH62WN6TLEEBULOTHEEA, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.956948251 (375h26m6.727589587s) [noise=true , address=77.90.31.21:4879 , piece_id=E2RFU72GA5KHGEHN6SZ2DRDPU3ANYV7TBTVY5TJFEOUNBUFR25LQ, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.965632647 (375h26m6.736273973s) [noise=true , address=77.90.31.19:4843 , piece_id=336ULA34A37WNNMZW74TXORG24UTHHDPRBBVEZBW44B3A4MVWUCA, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.967775456 (375h26m6.738416791s) [noise=true , address=77.90.31.16:4735 , piece_id=JJ2XV7T4VQX53VL3QEEHCPNMKVV6YF67YWNXYJRF5SP2YEAYW22A, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.972293731 (375h26m6.742935066s) [noise=true , address=77.90.31.10:4517 , piece_id=Z77S2VYE3Y6MNBJ3E2LKYGWIMY6IIEVO3LOMNDE3D57SUCGGPRUA, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.977592308 (375h26m6.748233643s) [noise=true , address=45.91.116.11:4117 , piece_id=F2K2VFZR4HJV3J45XB5X46XK7AYG7XQEP5N4JYBW4FAF3SOVRIPQ, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:31.991683597 (375h26m6.762324932s) [noise=true , address=45.91.116.7:4009 , piece_id=XXGMERGTXCXZ5KHY26ZTXDO5DISVXLCALRBKG6XKOILKKYUQO5EA, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:46.842204131 (375h26m21.612845466s) [noise=true , address=104.36.82.11:4114 , piece_id=7S6NXTZY7RPMFDYHUBC4A6G5JHOYWJRH455H5PYMU32FIE4KXQ2A, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:56.981597146 (375h26m31.752238481s) [noise=true , address=104.36.82.16:4294 , piece_id=YDM4PBG7FQMNILDJDTPH3EOLXKAZ3FULLFUTYFUOHPCSM5ICGR4Q, seg_pos={0 0}, upload=300357]: Done uploading piece. err: <nil> Jul 9 00:36:56.981622244 (375h26m31.752263579s) [seg_pos={0 0}, upload=300357]: Segment reached optimal threshold of 13 pieces. Jul 9 00:36:56.981704171 (375h26m31.752345506s) [noise=true , address=104.36.82.8:4972 , piece_id=3WE5NH4IDV2YUI6Q2C3FGOUADXIPCTP7DIJ32MSAHASWB4HBQOHQ, seg_pos={0 0}, upload=300357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 9 00:36:56.981751811 (375h26m31.752393147s) [noise=true , address=104.36.82.8:4970 , piece_id=XRIL73IEWUKUO73EU6457GO7YVIUWB34XY5Y227K26OOAYX7JT6Q, seg_pos={0 0}, upload=300357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: expected piece hash; context canceled; EOF Jul 9 00:36:56.981774735 (375h26m31.752416071s) [noise=true , address=104.36.82.4:4801 , piece_id=V4MBV5ZRQR32IOV2L7NYAQKRJAKTE3DZOKDWZO6DP3KI26N7ES3A, seg_pos={0 0}, upload=300357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ): protocol: expected piece hash; context canceled; EOF Jul 9 00:36:56.981823259 (375h26m31.752464604s) [seg_pos={0 0}, upload=300357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 00:36:56.981898523 (375h26m31.752539848s) [upload=300357]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 00:36:56.981903031 (375h26m31.752544356s) [upload=300357]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 00:36:57.283480701 (375h26m32.054122026s) [upload=300357]: Done uploading segments... Jul 9 00:36:57.283499177 (375h26m32.054140502s) [upload=300357]: Upload finished. err: <nil> Duration: 5.76506178s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)