Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X...............................X...............] duration: 1.970513844s FAIL (2025-08-07 04:45:37 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.97054309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.978691522s FAIL (2025-08-07 04:45:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h; context canceled; EOF Aug 7 04:39:49.932182402 (14h30m57.870598251s) [noise=true , address=77.90.31.10:4518 , piece_id=K5QMDHMMPD7ZXAQGZIS5LZLCOWBI63ZU6GZWX3G25VWG25PBFRKQ, seg_pos={0 0}, upload=13910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: context canceled; context canceled Aug 7 04:39:49.932198332 (14h30m57.870614171s) [seg_pos={0 0}, upload=13910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 04:39:49.932274307 (14h30m57.870690157s) [upload=13910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 04:39:49.932278435 (14h30m57.870694274s) [upload=13910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 04:39:50.221668416 (14h30m58.160084265s) [upload=13910]: Done uploading segments... Aug 7 04:39:50.221697813 (14h30m58.160113662s) [upload=13910]: Upload finished. err: <nil> Aug 7 04:40:10.709430510 (14h31m18.647846359s) [upload=13918]: Starting upload Aug 7 04:40:10.709680849 (14h31m18.648096698s) [upload=13918]: Done starting upload Aug 7 04:40:10.709769640 (14h31m18.648185489s) [upload=13918]: Uploading segments... Aug 7 04:40:10.709969702 (14h31m18.648385541s) [upload=13918]: Got next segment. Inline: false Aug 7 04:40:10.709994119 (14h31m18.648409968s) [upload=13918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 04:40:10.709998968 (14h31m18.648414817s) [upload=13918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 04:40:11.158815479 (14h31m19.097231328s) [seg_pos={0 0}, upload=13918]: Begin upload segment... Aug 7 04:40:11.158866977 (14h31m19.097282826s) [seg_pos={0 0}, upload=13918]: Done begin upload segment. Aug 7 04:40:11.158887697 (14h31m19.097303546s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.158931621 (14h31m19.09734747s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159313842 (14h31m19.097729692s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159386721 (14h31m19.09780256s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159633684 (14h31m19.098049533s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159670775 (14h31m19.098086614s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159981179 (14h31m19.098397028s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160267346 (14h31m19.098683185s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160614340 (14h31m19.099030189s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160919214 (14h31m19.099335063s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161272660 (14h31m19.099688509s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161642818 (14h31m19.100058667s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162100544 (14h31m19.100516393s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162369919 (14h31m19.100785768s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162608575 (14h31m19.101024425s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162841281 (14h31m19.10125713s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:12.797556894 (14h31m20.735972743s) [upload=13918]: Next returned nil segment Aug 7 04:40:12.797574407 (14h31m20.735990256s) [upload=13918]: Waiting for error group managing segments... Aug 7 04:40:13.182882502 (14h31m21.121298351s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.209187759 (14h31m21.147603608s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.255208999 (14h31m21.193624848s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.259802397 (14h31m21.198218247s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.266402565 (14h31m21.204818414s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.278029680 (14h31m21.216445529s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.335622499 (14h31m21.274038358s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.347771934 (14h31m21.286187783s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.361574822 (14h31m21.299990671s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.373787768 (14h31m21.312203617s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.378327684 (14h31m21.316743533s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.411533224 (14h31m21.349949073s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:40.062032468 (14h31m48.000448317s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062067876 (14h31m48.000483725s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062076252 (14h31m48.000492101s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062085720 (14h31m48.000501569s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062105017 (14h31m48.000520866s) [seg_pos={0 0}, upload=13918]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 7 04:40:40.407955838 (14h31m48.346371677s) [upload=13918]: Done uploading segments... Aug 7 04:40:40.407976819 (14h31m48.346392668s) [upload=13918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.97872151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.759138587s FAIL (2025-08-07 04:45:17 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.759163254s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.039789977s FAIL (2025-08-07 04:44:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.039809605s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.7651867s FAIL (2025-08-07 04:44:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h; context canceled; EOF Aug 7 04:39:49.932182402 (14h30m57.870598251s) [noise=true , address=77.90.31.10:4518 , piece_id=K5QMDHMMPD7ZXAQGZIS5LZLCOWBI63ZU6GZWX3G25VWG25PBFRKQ, seg_pos={0 0}, upload=13910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: context canceled; context canceled Aug 7 04:39:49.932198332 (14h30m57.870614171s) [seg_pos={0 0}, upload=13910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 04:39:49.932274307 (14h30m57.870690157s) [upload=13910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 04:39:49.932278435 (14h30m57.870694274s) [upload=13910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 04:39:50.221668416 (14h30m58.160084265s) [upload=13910]: Done uploading segments... Aug 7 04:39:50.221697813 (14h30m58.160113662s) [upload=13910]: Upload finished. err: <nil> Aug 7 04:40:10.709430510 (14h31m18.647846359s) [upload=13918]: Starting upload Aug 7 04:40:10.709680849 (14h31m18.648096698s) [upload=13918]: Done starting upload Aug 7 04:40:10.709769640 (14h31m18.648185489s) [upload=13918]: Uploading segments... Aug 7 04:40:10.709969702 (14h31m18.648385541s) [upload=13918]: Got next segment. Inline: false Aug 7 04:40:10.709994119 (14h31m18.648409968s) [upload=13918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 04:40:10.709998968 (14h31m18.648414817s) [upload=13918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 04:40:11.158815479 (14h31m19.097231328s) [seg_pos={0 0}, upload=13918]: Begin upload segment... Aug 7 04:40:11.158866977 (14h31m19.097282826s) [seg_pos={0 0}, upload=13918]: Done begin upload segment. Aug 7 04:40:11.158887697 (14h31m19.097303546s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.158931621 (14h31m19.09734747s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159313842 (14h31m19.097729692s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159386721 (14h31m19.09780256s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159633684 (14h31m19.098049533s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159670775 (14h31m19.098086614s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159981179 (14h31m19.098397028s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160267346 (14h31m19.098683185s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160614340 (14h31m19.099030189s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160919214 (14h31m19.099335063s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161272660 (14h31m19.099688509s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161642818 (14h31m19.100058667s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162100544 (14h31m19.100516393s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162369919 (14h31m19.100785768s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162608575 (14h31m19.101024425s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162841281 (14h31m19.10125713s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:12.797556894 (14h31m20.735972743s) [upload=13918]: Next returned nil segment Aug 7 04:40:12.797574407 (14h31m20.735990256s) [upload=13918]: Waiting for error group managing segments... Aug 7 04:40:13.182882502 (14h31m21.121298351s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.209187759 (14h31m21.147603608s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.255208999 (14h31m21.193624848s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.259802397 (14h31m21.198218247s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.266402565 (14h31m21.204818414s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.278029680 (14h31m21.216445529s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.335622499 (14h31m21.274038358s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.347771934 (14h31m21.286187783s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.361574822 (14h31m21.299990671s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.373787768 (14h31m21.312203617s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.378327684 (14h31m21.316743533s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.411533224 (14h31m21.349949073s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:40.062032468 (14h31m48.000448317s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062067876 (14h31m48.000483725s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062076252 (14h31m48.000492101s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062085720 (14h31m48.000501569s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062105017 (14h31m48.000520866s) [seg_pos={0 0}, upload=13918]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 7 04:40:40.407955838 (14h31m48.346371677s) [upload=13918]: Done uploading segments... Aug 7 04:40:40.407976819 (14h31m48.346392668s) [upload=13918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.765212759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X...............................X..............] duration: 1.957167147s FAIL (2025-08-07 04:44:33 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.957197055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.544214915s FAIL (2025-08-07 04:44:31 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.544233812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.485537254s FAIL (2025-08-07 04:44:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.485556421s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....................................................X.....] duration: 6.358929166s FAIL (2025-08-07 04:43:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h; context canceled; EOF Aug 7 04:39:49.932182402 (14h30m57.870598251s) [noise=true , address=77.90.31.10:4518 , piece_id=K5QMDHMMPD7ZXAQGZIS5LZLCOWBI63ZU6GZWX3G25VWG25PBFRKQ, seg_pos={0 0}, upload=13910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: context canceled; context canceled Aug 7 04:39:49.932198332 (14h30m57.870614171s) [seg_pos={0 0}, upload=13910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 04:39:49.932274307 (14h30m57.870690157s) [upload=13910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 04:39:49.932278435 (14h30m57.870694274s) [upload=13910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 04:39:50.221668416 (14h30m58.160084265s) [upload=13910]: Done uploading segments... Aug 7 04:39:50.221697813 (14h30m58.160113662s) [upload=13910]: Upload finished. err: <nil> Aug 7 04:40:10.709430510 (14h31m18.647846359s) [upload=13918]: Starting upload Aug 7 04:40:10.709680849 (14h31m18.648096698s) [upload=13918]: Done starting upload Aug 7 04:40:10.709769640 (14h31m18.648185489s) [upload=13918]: Uploading segments... Aug 7 04:40:10.709969702 (14h31m18.648385541s) [upload=13918]: Got next segment. Inline: false Aug 7 04:40:10.709994119 (14h31m18.648409968s) [upload=13918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 04:40:10.709998968 (14h31m18.648414817s) [upload=13918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 04:40:11.158815479 (14h31m19.097231328s) [seg_pos={0 0}, upload=13918]: Begin upload segment... Aug 7 04:40:11.158866977 (14h31m19.097282826s) [seg_pos={0 0}, upload=13918]: Done begin upload segment. Aug 7 04:40:11.158887697 (14h31m19.097303546s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.158931621 (14h31m19.09734747s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159313842 (14h31m19.097729692s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159386721 (14h31m19.09780256s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159633684 (14h31m19.098049533s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159670775 (14h31m19.098086614s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159981179 (14h31m19.098397028s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160267346 (14h31m19.098683185s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160614340 (14h31m19.099030189s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160919214 (14h31m19.099335063s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161272660 (14h31m19.099688509s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161642818 (14h31m19.100058667s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162100544 (14h31m19.100516393s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162369919 (14h31m19.100785768s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162608575 (14h31m19.101024425s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162841281 (14h31m19.10125713s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:12.797556894 (14h31m20.735972743s) [upload=13918]: Next returned nil segment Aug 7 04:40:12.797574407 (14h31m20.735990256s) [upload=13918]: Waiting for error group managing segments... Aug 7 04:40:13.182882502 (14h31m21.121298351s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.209187759 (14h31m21.147603608s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.255208999 (14h31m21.193624848s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.259802397 (14h31m21.198218247s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.266402565 (14h31m21.204818414s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.278029680 (14h31m21.216445529s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.335622499 (14h31m21.274038358s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.347771934 (14h31m21.286187783s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.361574822 (14h31m21.299990671s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.373787768 (14h31m21.312203617s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.378327684 (14h31m21.316743533s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.411533224 (14h31m21.349949073s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:40.062032468 (14h31m48.000448317s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062067876 (14h31m48.000483725s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062076252 (14h31m48.000492101s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062085720 (14h31m48.000501569s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062105017 (14h31m48.000520866s) [seg_pos={0 0}, upload=13918]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 7 04:40:40.407955838 (14h31m48.346371677s) [upload=13918]: Done uploading segments... Aug 7 04:40:40.407976819 (14h31m48.346392668s) [upload=13918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.358960126s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.567405981s FAIL (2025-08-07 04:43:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.567588219s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.85535094s FAIL (2025-08-07 04:43:39 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.85538254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X...............................X.............] duration: 1.866533512s FAIL (2025-08-07 04:43:20 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.866555236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....................................................X....] duration: 5.987044113s FAIL (2025-08-07 04:43:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h; context canceled; EOF Aug 7 04:39:49.932182402 (14h30m57.870598251s) [noise=true , address=77.90.31.10:4518 , piece_id=K5QMDHMMPD7ZXAQGZIS5LZLCOWBI63ZU6GZWX3G25VWG25PBFRKQ, seg_pos={0 0}, upload=13910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: context canceled; context canceled Aug 7 04:39:49.932198332 (14h30m57.870614171s) [seg_pos={0 0}, upload=13910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 04:39:49.932274307 (14h30m57.870690157s) [upload=13910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 04:39:49.932278435 (14h30m57.870694274s) [upload=13910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 04:39:50.221668416 (14h30m58.160084265s) [upload=13910]: Done uploading segments... Aug 7 04:39:50.221697813 (14h30m58.160113662s) [upload=13910]: Upload finished. err: <nil> Aug 7 04:40:10.709430510 (14h31m18.647846359s) [upload=13918]: Starting upload Aug 7 04:40:10.709680849 (14h31m18.648096698s) [upload=13918]: Done starting upload Aug 7 04:40:10.709769640 (14h31m18.648185489s) [upload=13918]: Uploading segments... Aug 7 04:40:10.709969702 (14h31m18.648385541s) [upload=13918]: Got next segment. Inline: false Aug 7 04:40:10.709994119 (14h31m18.648409968s) [upload=13918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 04:40:10.709998968 (14h31m18.648414817s) [upload=13918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 04:40:11.158815479 (14h31m19.097231328s) [seg_pos={0 0}, upload=13918]: Begin upload segment... Aug 7 04:40:11.158866977 (14h31m19.097282826s) [seg_pos={0 0}, upload=13918]: Done begin upload segment. Aug 7 04:40:11.158887697 (14h31m19.097303546s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.158931621 (14h31m19.09734747s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159313842 (14h31m19.097729692s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159386721 (14h31m19.09780256s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159633684 (14h31m19.098049533s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159670775 (14h31m19.098086614s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159981179 (14h31m19.098397028s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160267346 (14h31m19.098683185s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160614340 (14h31m19.099030189s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160919214 (14h31m19.099335063s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161272660 (14h31m19.099688509s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161642818 (14h31m19.100058667s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162100544 (14h31m19.100516393s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162369919 (14h31m19.100785768s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162608575 (14h31m19.101024425s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162841281 (14h31m19.10125713s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:12.797556894 (14h31m20.735972743s) [upload=13918]: Next returned nil segment Aug 7 04:40:12.797574407 (14h31m20.735990256s) [upload=13918]: Waiting for error group managing segments... Aug 7 04:40:13.182882502 (14h31m21.121298351s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.209187759 (14h31m21.147603608s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.255208999 (14h31m21.193624848s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.259802397 (14h31m21.198218247s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.266402565 (14h31m21.204818414s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.278029680 (14h31m21.216445529s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.335622499 (14h31m21.274038358s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.347771934 (14h31m21.286187783s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.361574822 (14h31m21.299990671s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.373787768 (14h31m21.312203617s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.378327684 (14h31m21.316743533s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.411533224 (14h31m21.349949073s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:40.062032468 (14h31m48.000448317s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062067876 (14h31m48.000483725s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062076252 (14h31m48.000492101s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062085720 (14h31m48.000501569s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062105017 (14h31m48.000520866s) [seg_pos={0 0}, upload=13918]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 7 04:40:40.407955838 (14h31m48.346371677s) [upload=13918]: Done uploading segments... Aug 7 04:40:40.407976819 (14h31m48.346392668s) [upload=13918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.98707348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X...............................X............] duration: 1.890057992s FAIL (2025-08-07 04:42:50 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.89008344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.598097029s FAIL (2025-08-07 04:42:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.598132458s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.97013444s FAIL (2025-08-07 04:42:38 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.970156702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....................................................X...] duration: 5.790817784s FAIL (2025-08-07 04:42:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h; context canceled; EOF Aug 7 04:39:49.932182402 (14h30m57.870598251s) [noise=true , address=77.90.31.10:4518 , piece_id=K5QMDHMMPD7ZXAQGZIS5LZLCOWBI63ZU6GZWX3G25VWG25PBFRKQ, seg_pos={0 0}, upload=13910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: context canceled; context canceled Aug 7 04:39:49.932198332 (14h30m57.870614171s) [seg_pos={0 0}, upload=13910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 04:39:49.932274307 (14h30m57.870690157s) [upload=13910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 04:39:49.932278435 (14h30m57.870694274s) [upload=13910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 04:39:50.221668416 (14h30m58.160084265s) [upload=13910]: Done uploading segments... Aug 7 04:39:50.221697813 (14h30m58.160113662s) [upload=13910]: Upload finished. err: <nil> Aug 7 04:40:10.709430510 (14h31m18.647846359s) [upload=13918]: Starting upload Aug 7 04:40:10.709680849 (14h31m18.648096698s) [upload=13918]: Done starting upload Aug 7 04:40:10.709769640 (14h31m18.648185489s) [upload=13918]: Uploading segments... Aug 7 04:40:10.709969702 (14h31m18.648385541s) [upload=13918]: Got next segment. Inline: false Aug 7 04:40:10.709994119 (14h31m18.648409968s) [upload=13918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 04:40:10.709998968 (14h31m18.648414817s) [upload=13918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 04:40:11.158815479 (14h31m19.097231328s) [seg_pos={0 0}, upload=13918]: Begin upload segment... Aug 7 04:40:11.158866977 (14h31m19.097282826s) [seg_pos={0 0}, upload=13918]: Done begin upload segment. Aug 7 04:40:11.158887697 (14h31m19.097303546s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.158931621 (14h31m19.09734747s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159313842 (14h31m19.097729692s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159386721 (14h31m19.09780256s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159633684 (14h31m19.098049533s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159670775 (14h31m19.098086614s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159981179 (14h31m19.098397028s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160267346 (14h31m19.098683185s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160614340 (14h31m19.099030189s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160919214 (14h31m19.099335063s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161272660 (14h31m19.099688509s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161642818 (14h31m19.100058667s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162100544 (14h31m19.100516393s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162369919 (14h31m19.100785768s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162608575 (14h31m19.101024425s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162841281 (14h31m19.10125713s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:12.797556894 (14h31m20.735972743s) [upload=13918]: Next returned nil segment Aug 7 04:40:12.797574407 (14h31m20.735990256s) [upload=13918]: Waiting for error group managing segments... Aug 7 04:40:13.182882502 (14h31m21.121298351s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.209187759 (14h31m21.147603608s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.255208999 (14h31m21.193624848s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.259802397 (14h31m21.198218247s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.266402565 (14h31m21.204818414s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.278029680 (14h31m21.216445529s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.335622499 (14h31m21.274038358s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.347771934 (14h31m21.286187783s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.361574822 (14h31m21.299990671s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.373787768 (14h31m21.312203617s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.378327684 (14h31m21.316743533s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.411533224 (14h31m21.349949073s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:40.062032468 (14h31m48.000448317s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062067876 (14h31m48.000483725s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062076252 (14h31m48.000492101s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062085720 (14h31m48.000501569s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062105017 (14h31m48.000520866s) [seg_pos={0 0}, upload=13918]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 7 04:40:40.407955838 (14h31m48.346371677s) [upload=13918]: Done uploading segments... Aug 7 04:40:40.407976819 (14h31m48.346392668s) [upload=13918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.790856108s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....................................................X..] duration: 5.920967563s FAIL (2025-08-07 04:42:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h; context canceled; EOF Aug 7 04:39:49.932182402 (14h30m57.870598251s) [noise=true , address=77.90.31.10:4518 , piece_id=K5QMDHMMPD7ZXAQGZIS5LZLCOWBI63ZU6GZWX3G25VWG25PBFRKQ, seg_pos={0 0}, upload=13910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: context canceled; context canceled Aug 7 04:39:49.932198332 (14h30m57.870614171s) [seg_pos={0 0}, upload=13910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 04:39:49.932274307 (14h30m57.870690157s) [upload=13910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 04:39:49.932278435 (14h30m57.870694274s) [upload=13910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 04:39:50.221668416 (14h30m58.160084265s) [upload=13910]: Done uploading segments... Aug 7 04:39:50.221697813 (14h30m58.160113662s) [upload=13910]: Upload finished. err: <nil> Aug 7 04:40:10.709430510 (14h31m18.647846359s) [upload=13918]: Starting upload Aug 7 04:40:10.709680849 (14h31m18.648096698s) [upload=13918]: Done starting upload Aug 7 04:40:10.709769640 (14h31m18.648185489s) [upload=13918]: Uploading segments... Aug 7 04:40:10.709969702 (14h31m18.648385541s) [upload=13918]: Got next segment. Inline: false Aug 7 04:40:10.709994119 (14h31m18.648409968s) [upload=13918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 04:40:10.709998968 (14h31m18.648414817s) [upload=13918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 04:40:11.158815479 (14h31m19.097231328s) [seg_pos={0 0}, upload=13918]: Begin upload segment... Aug 7 04:40:11.158866977 (14h31m19.097282826s) [seg_pos={0 0}, upload=13918]: Done begin upload segment. Aug 7 04:40:11.158887697 (14h31m19.097303546s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.158931621 (14h31m19.09734747s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159313842 (14h31m19.097729692s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159386721 (14h31m19.09780256s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159633684 (14h31m19.098049533s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159670775 (14h31m19.098086614s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159981179 (14h31m19.098397028s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160267346 (14h31m19.098683185s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160614340 (14h31m19.099030189s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160919214 (14h31m19.099335063s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161272660 (14h31m19.099688509s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161642818 (14h31m19.100058667s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162100544 (14h31m19.100516393s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162369919 (14h31m19.100785768s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162608575 (14h31m19.101024425s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162841281 (14h31m19.10125713s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:12.797556894 (14h31m20.735972743s) [upload=13918]: Next returned nil segment Aug 7 04:40:12.797574407 (14h31m20.735990256s) [upload=13918]: Waiting for error group managing segments... Aug 7 04:40:13.182882502 (14h31m21.121298351s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.209187759 (14h31m21.147603608s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.255208999 (14h31m21.193624848s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.259802397 (14h31m21.198218247s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.266402565 (14h31m21.204818414s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.278029680 (14h31m21.216445529s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.335622499 (14h31m21.274038358s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.347771934 (14h31m21.286187783s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.361574822 (14h31m21.299990671s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.373787768 (14h31m21.312203617s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.378327684 (14h31m21.316743533s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.411533224 (14h31m21.349949073s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:40.062032468 (14h31m48.000448317s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062067876 (14h31m48.000483725s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062076252 (14h31m48.000492101s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062085720 (14h31m48.000501569s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062105017 (14h31m48.000520866s) [seg_pos={0 0}, upload=13918]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 7 04:40:40.407955838 (14h31m48.346371677s) [upload=13918]: Done uploading segments... Aug 7 04:40:40.407976819 (14h31m48.346392668s) [upload=13918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.921002261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X...............................X...........] duration: 2.211296065s FAIL (2025-08-07 04:42:08 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 2.211323015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.57769888s FAIL (2025-08-07 04:42:07 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.577728194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.983247698s FAIL (2025-08-07 04:41:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.983283096s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.239611295s FAIL (2025-08-07 04:41:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.239634722s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X...............................X..........] duration: 2.119697866s FAIL (2025-08-07 04:41:04 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 2.119729696s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....................................................X.] duration: 5.92742702s FAIL (2025-08-07 04:41:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h; context canceled; EOF Aug 7 04:39:49.932182402 (14h30m57.870598251s) [noise=true , address=77.90.31.10:4518 , piece_id=K5QMDHMMPD7ZXAQGZIS5LZLCOWBI63ZU6GZWX3G25VWG25PBFRKQ, seg_pos={0 0}, upload=13910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: context canceled; context canceled Aug 7 04:39:49.932198332 (14h30m57.870614171s) [seg_pos={0 0}, upload=13910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 04:39:49.932274307 (14h30m57.870690157s) [upload=13910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 04:39:49.932278435 (14h30m57.870694274s) [upload=13910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 04:39:50.221668416 (14h30m58.160084265s) [upload=13910]: Done uploading segments... Aug 7 04:39:50.221697813 (14h30m58.160113662s) [upload=13910]: Upload finished. err: <nil> Aug 7 04:40:10.709430510 (14h31m18.647846359s) [upload=13918]: Starting upload Aug 7 04:40:10.709680849 (14h31m18.648096698s) [upload=13918]: Done starting upload Aug 7 04:40:10.709769640 (14h31m18.648185489s) [upload=13918]: Uploading segments... Aug 7 04:40:10.709969702 (14h31m18.648385541s) [upload=13918]: Got next segment. Inline: false Aug 7 04:40:10.709994119 (14h31m18.648409968s) [upload=13918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 04:40:10.709998968 (14h31m18.648414817s) [upload=13918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 04:40:11.158815479 (14h31m19.097231328s) [seg_pos={0 0}, upload=13918]: Begin upload segment... Aug 7 04:40:11.158866977 (14h31m19.097282826s) [seg_pos={0 0}, upload=13918]: Done begin upload segment. Aug 7 04:40:11.158887697 (14h31m19.097303546s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.158931621 (14h31m19.09734747s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159313842 (14h31m19.097729692s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159386721 (14h31m19.09780256s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159633684 (14h31m19.098049533s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159670775 (14h31m19.098086614s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159981179 (14h31m19.098397028s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160267346 (14h31m19.098683185s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160614340 (14h31m19.099030189s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160919214 (14h31m19.099335063s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161272660 (14h31m19.099688509s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161642818 (14h31m19.100058667s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162100544 (14h31m19.100516393s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162369919 (14h31m19.100785768s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162608575 (14h31m19.101024425s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162841281 (14h31m19.10125713s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:12.797556894 (14h31m20.735972743s) [upload=13918]: Next returned nil segment Aug 7 04:40:12.797574407 (14h31m20.735990256s) [upload=13918]: Waiting for error group managing segments... Aug 7 04:40:13.182882502 (14h31m21.121298351s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.209187759 (14h31m21.147603608s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.255208999 (14h31m21.193624848s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.259802397 (14h31m21.198218247s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.266402565 (14h31m21.204818414s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.278029680 (14h31m21.216445529s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.335622499 (14h31m21.274038358s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.347771934 (14h31m21.286187783s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.361574822 (14h31m21.299990671s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.373787768 (14h31m21.312203617s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.378327684 (14h31m21.316743533s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.411533224 (14h31m21.349949073s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:40.062032468 (14h31m48.000448317s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062067876 (14h31m48.000483725s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062076252 (14h31m48.000492101s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062085720 (14h31m48.000501569s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062105017 (14h31m48.000520866s) [seg_pos={0 0}, upload=13918]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 7 04:40:40.407955838 (14h31m48.346371677s) [upload=13918]: Done uploading segments... Aug 7 04:40:40.407976819 (14h31m48.346392668s) [upload=13918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.927454754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.701200355s FAIL (2025-08-07 04:40:55 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.701224844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001275547s FAIL (2025-08-07 04:40:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001293992s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....................................................X] duration: 30.347055066s FAIL (2025-08-07 04:40:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== h; context canceled; EOF Aug 7 04:39:49.932182402 (14h30m57.870598251s) [noise=true , address=77.90.31.10:4518 , piece_id=K5QMDHMMPD7ZXAQGZIS5LZLCOWBI63ZU6GZWX3G25VWG25PBFRKQ, seg_pos={0 0}, upload=13910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: context canceled; context canceled Aug 7 04:39:49.932198332 (14h30m57.870614171s) [seg_pos={0 0}, upload=13910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 04:39:49.932274307 (14h30m57.870690157s) [upload=13910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 04:39:49.932278435 (14h30m57.870694274s) [upload=13910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 04:39:50.221668416 (14h30m58.160084265s) [upload=13910]: Done uploading segments... Aug 7 04:39:50.221697813 (14h30m58.160113662s) [upload=13910]: Upload finished. err: <nil> Aug 7 04:40:10.709430510 (14h31m18.647846359s) [upload=13918]: Starting upload Aug 7 04:40:10.709680849 (14h31m18.648096698s) [upload=13918]: Done starting upload Aug 7 04:40:10.709769640 (14h31m18.648185489s) [upload=13918]: Uploading segments... Aug 7 04:40:10.709969702 (14h31m18.648385541s) [upload=13918]: Got next segment. Inline: false Aug 7 04:40:10.709994119 (14h31m18.648409968s) [upload=13918]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 04:40:10.709998968 (14h31m18.648414817s) [upload=13918]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 04:40:11.158815479 (14h31m19.097231328s) [seg_pos={0 0}, upload=13918]: Begin upload segment... Aug 7 04:40:11.158866977 (14h31m19.097282826s) [seg_pos={0 0}, upload=13918]: Done begin upload segment. Aug 7 04:40:11.158887697 (14h31m19.097303546s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.158931621 (14h31m19.09734747s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159313842 (14h31m19.097729692s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159386721 (14h31m19.09780256s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159633684 (14h31m19.098049533s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159670775 (14h31m19.098086614s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.159981179 (14h31m19.098397028s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160267346 (14h31m19.098683185s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160614340 (14h31m19.099030189s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.160919214 (14h31m19.099335063s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161272660 (14h31m19.099688509s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.161642818 (14h31m19.100058667s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162100544 (14h31m19.100516393s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162369919 (14h31m19.100785768s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162608575 (14h31m19.101024425s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:11.162841281 (14h31m19.10125713s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Uploading piece... Aug 7 04:40:12.797556894 (14h31m20.735972743s) [upload=13918]: Next returned nil segment Aug 7 04:40:12.797574407 (14h31m20.735990256s) [upload=13918]: Waiting for error group managing segments... Aug 7 04:40:13.182882502 (14h31m21.121298351s) [noise=true , address=45.91.116.11:4118 , piece_id=JNCRQLX5AMDOPT5GZ37ZX5ET5ZQNMM3GXONLQBIBIXSJM2F7OIJQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.209187759 (14h31m21.147603608s) [noise=true , address=45.91.116.4:4940 , piece_id=G3VJSBWDGP2QH6RTSQZ62J5J4DO6EUTOROT4FDGVK54QRYJGANLQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.255208999 (14h31m21.193624848s) [noise=true , address=45.91.116.5:4299 , piece_id=LJP4XUE5ERJ4MD4ZNGNHHGFB2WRMWJQNKRSXFJU2LS3KQVMJDI5Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.259802397 (14h31m21.198218247s) [noise=true , address=45.91.116.7:4009 , piece_id=ACQR4SMFVTC6B5QC22AJZAYXADNEFU5RIO7QJ4V6LDZLANZWAPAQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.266402565 (14h31m21.204818414s) [noise=true , address=45.91.116.6:4975 , piece_id=DNZR7UZC55XEKKHDAQSIFXVBLYF55MZ7N3KZ5F7FXPY4EGE7HOGQ, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.278029680 (14h31m21.216445529s) [noise=true , address=104.36.82.10:4078 , piece_id=BVIS4ADO3O4SN2Q6BY7KGT5WBAGEM2ZSOCWVOIIQSEAFR35UAI7Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.335622499 (14h31m21.274038358s) [noise=true , address=45.91.116.10:4082 , piece_id=2PWQKVLKQR22YFJWLNH5FS2R45ZKIEGYG5SN3WN7VIOLVIE7N7YA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.347771934 (14h31m21.286187783s) [noise=true , address=45.91.116.13:4154 , piece_id=R3TC4YALR7XMMVIKMH6D4IQKT44D35WPMDXI42QX2IZ7FNKJNGBA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.361574822 (14h31m21.299990671s) [noise=true , address=45.91.116.8:4048 , piece_id=ZYX7Y2KWPJ6MMXFEKNYQX5ZFJDAQEGGSIS2CP46MACUR4LCHHIYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.373787768 (14h31m21.312203617s) [noise=true , address=104.36.82.5:4839 , piece_id=AHOLJQHRNJBODKWNKYRTWPJLSJUQBCKUBDKA44CI5VJTXI2ADKQA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.378327684 (14h31m21.316743533s) [noise=true , address=104.36.82.3:4767 , piece_id=6HLEJVT5QJWR252CNZETFEYW7D7244557CJ5R22K45JBMHCYAZ6Q, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:13.411533224 (14h31m21.349949073s) [noise=true , address=104.36.82.16:4294 , piece_id=DQU2PDXPQWOLX5LQS25X6ORYXLI5TY6JV73RMWPRXARCS55KAZOA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: <nil> Aug 7 04:40:40.062032468 (14h31m48.000448317s) [noise=true , address=77.90.31.13:4628 , piece_id=SI2QYS4WOR4ERQHTMVPWPUL2UMASQCRZSHFKMXLADMZ2RQCIEVAA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062067876 (14h31m48.000483725s) [noise=true , address=77.90.31.11:4555 , piece_id=YOUMGI7FAYQGRUVKPIESESXVK7WEWDWU5X72OTH66UW22EPO7SRA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062076252 (14h31m48.000492101s) [noise=true , address=77.90.31.12:4590 , piece_id=2Z7DTOKO27RVEQ45FV6NL34EQWKLALBEKCTQSWNJHABWJRL67VKA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062085720 (14h31m48.000501569s) [noise=true , address=77.90.31.18:4807 , piece_id=XHKKWFKZCSIGM42QGORZ3K5DAQ2YRIT2AHIHECYOUZ327CAVONYA, seg_pos={0 0}, upload=13918]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 04:40:40.062105017 (14h31m48.000520866s) [seg_pos={0 0}, upload=13918]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 7 04:40:40.407955838 (14h31m48.346371677s) [upload=13918]: Done uploading segments... Aug 7 04:40:40.407976819 (14h31m48.346392668s) [upload=13918]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.347081247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.871380934s FAIL (2025-08-07 04:40:20 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.871405751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.789401919s FAIL (2025-08-07 04:40:02 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.789430644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.786182067s FAIL (2025-08-07 04:39:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== WatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136236350 (13h49m24.074652199s) [noise=true , address=77.90.31.10:4519 , piece_id=XI4WMZRV2Y4HJ2FEJN6G6ZQSIXKPUI2ZSZQAZRW4M4XRTMNGIHPQ, seg_pos={0 0}, upload=13245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136320912 (13h49m24.074736771s) [seg_pos={0 0}, upload=13245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:16.136399292 (13h49m24.074815141s) [upload=13245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:16.136403420 (13h49m24.074819269s) [upload=13245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:16.427080155 (13h49m24.365496004s) [upload=13245]: Done uploading segments... Aug 7 03:58:16.427097879 (13h49m24.365513729s) [upload=13245]: Upload finished. err: <nil> Aug 7 03:58:30.184956648 (13h49m38.123372497s) [upload=13250]: Starting upload Aug 7 03:58:30.185004009 (13h49m38.123419848s) [upload=13250]: Done starting upload Aug 7 03:58:30.185205905 (13h49m38.123621744s) [upload=13250]: Uploading segments... Aug 7 03:58:30.185236314 (13h49m38.123652163s) [upload=13250]: Got next segment. Inline: false Aug 7 03:58:30.185257183 (13h49m38.123673022s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 03:58:30.185260930 (13h49m38.123676779s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 03:58:30.454001938 (13h49m38.392417787s) [seg_pos={0 0}, upload=13250]: Begin upload segment... Aug 7 03:58:30.454044699 (13h49m38.392460548s) [seg_pos={0 0}, upload=13250]: Done begin upload segment. Aug 7 03:58:30.454066982 (13h49m38.392482831s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454100586 (13h49m38.392516435s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454427672 (13h49m38.392843511s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454588760 (13h49m38.393004609s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454692479 (13h49m38.393108328s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454908060 (13h49m38.393323909s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454963637 (13h49m38.393379486s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455232281 (13h49m38.39364813s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455255596 (13h49m38.393671445s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455479334 (13h49m38.393895183s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455506566 (13h49m38.393922405s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455787894 (13h49m38.394203743s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455835545 (13h49m38.394251384s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456019998 (13h49m38.394435848s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456223638 (13h49m38.394639487s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.457172833 (13h49m38.395588682s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:32.206529945 (13h49m40.144945785s) [upload=13250]: Next returned nil segment Aug 7 03:58:32.206545354 (13h49m40.144961203s) [upload=13250]: Waiting for error group managing segments... Aug 7 03:58:32.482974069 (13h49m40.421389908s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.546641532 (13h49m40.485057371s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.547859872 (13h49m40.486275711s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.555284067 (13h49m40.493699906s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.561331598 (13h49m40.499747447s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.562139544 (13h49m40.500555393s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.566721550 (13h49m40.505137399s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.567470363 (13h49m40.505886212s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.576776648 (13h49m40.515192497s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.580990250 (13h49m40.519406099s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.584837120 (13h49m40.523252959s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.585356173 (13h49m40.523772042s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601057214 (13h49m40.539473053s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601086330 (13h49m40.539502219s) [seg_pos={0 0}, upload=13250]: Segment reached optimal threshold of 13 pieces. Aug 7 03:58:32.601212661 (13h49m40.53962851s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601285240 (13h49m40.539701089s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601415780 (13h49m40.539831629s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601463891 (13h49m40.53987973s) [seg_pos={0 0}, upload=13250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:32.601549495 (13h49m40.539965334s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:32.601553784 (13h49m40.539969633s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:32.906871209 (13h49m40.845287058s) [upload=13250]: Done uploading segments... Aug 7 03:58:32.906889855 (13h49m40.845305704s) [upload=13250]: Upload finished. err: <nil> Duration: 5.786207985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X...............................X.........] duration: 1.871242406s FAIL (2025-08-07 04:39:50 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.871271151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.697954668s FAIL (2025-08-07 04:39:29 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.698066412s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.635665136s FAIL (2025-08-07 04:38:54 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.635689904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.069151045s FAIL (2025-08-07 04:38:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== WatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136236350 (13h49m24.074652199s) [noise=true , address=77.90.31.10:4519 , piece_id=XI4WMZRV2Y4HJ2FEJN6G6ZQSIXKPUI2ZSZQAZRW4M4XRTMNGIHPQ, seg_pos={0 0}, upload=13245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136320912 (13h49m24.074736771s) [seg_pos={0 0}, upload=13245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:16.136399292 (13h49m24.074815141s) [upload=13245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:16.136403420 (13h49m24.074819269s) [upload=13245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:16.427080155 (13h49m24.365496004s) [upload=13245]: Done uploading segments... Aug 7 03:58:16.427097879 (13h49m24.365513729s) [upload=13245]: Upload finished. err: <nil> Aug 7 03:58:30.184956648 (13h49m38.123372497s) [upload=13250]: Starting upload Aug 7 03:58:30.185004009 (13h49m38.123419848s) [upload=13250]: Done starting upload Aug 7 03:58:30.185205905 (13h49m38.123621744s) [upload=13250]: Uploading segments... Aug 7 03:58:30.185236314 (13h49m38.123652163s) [upload=13250]: Got next segment. Inline: false Aug 7 03:58:30.185257183 (13h49m38.123673022s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 03:58:30.185260930 (13h49m38.123676779s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 03:58:30.454001938 (13h49m38.392417787s) [seg_pos={0 0}, upload=13250]: Begin upload segment... Aug 7 03:58:30.454044699 (13h49m38.392460548s) [seg_pos={0 0}, upload=13250]: Done begin upload segment. Aug 7 03:58:30.454066982 (13h49m38.392482831s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454100586 (13h49m38.392516435s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454427672 (13h49m38.392843511s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454588760 (13h49m38.393004609s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454692479 (13h49m38.393108328s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454908060 (13h49m38.393323909s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454963637 (13h49m38.393379486s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455232281 (13h49m38.39364813s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455255596 (13h49m38.393671445s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455479334 (13h49m38.393895183s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455506566 (13h49m38.393922405s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455787894 (13h49m38.394203743s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455835545 (13h49m38.394251384s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456019998 (13h49m38.394435848s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456223638 (13h49m38.394639487s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.457172833 (13h49m38.395588682s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:32.206529945 (13h49m40.144945785s) [upload=13250]: Next returned nil segment Aug 7 03:58:32.206545354 (13h49m40.144961203s) [upload=13250]: Waiting for error group managing segments... Aug 7 03:58:32.482974069 (13h49m40.421389908s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.546641532 (13h49m40.485057371s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.547859872 (13h49m40.486275711s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.555284067 (13h49m40.493699906s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.561331598 (13h49m40.499747447s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.562139544 (13h49m40.500555393s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.566721550 (13h49m40.505137399s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.567470363 (13h49m40.505886212s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.576776648 (13h49m40.515192497s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.580990250 (13h49m40.519406099s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.584837120 (13h49m40.523252959s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.585356173 (13h49m40.523772042s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601057214 (13h49m40.539473053s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601086330 (13h49m40.539502219s) [seg_pos={0 0}, upload=13250]: Segment reached optimal threshold of 13 pieces. Aug 7 03:58:32.601212661 (13h49m40.53962851s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601285240 (13h49m40.539701089s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601415780 (13h49m40.539831629s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601463891 (13h49m40.53987973s) [seg_pos={0 0}, upload=13250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:32.601549495 (13h49m40.539965334s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:32.601553784 (13h49m40.539969633s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:32.906871209 (13h49m40.845287058s) [upload=13250]: Done uploading segments... Aug 7 03:58:32.906889855 (13h49m40.845305704s) [upload=13250]: Upload finished. err: <nil> Duration: 6.069176293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X...............................X........] duration: 1.855640904s FAIL (2025-08-07 04:38:46 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.855668036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X...............................X.......] duration: 1.868806242s FAIL (2025-08-07 04:38:22 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.868848252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.917400157s FAIL (2025-08-07 04:38:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== WatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136236350 (13h49m24.074652199s) [noise=true , address=77.90.31.10:4519 , piece_id=XI4WMZRV2Y4HJ2FEJN6G6ZQSIXKPUI2ZSZQAZRW4M4XRTMNGIHPQ, seg_pos={0 0}, upload=13245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136320912 (13h49m24.074736771s) [seg_pos={0 0}, upload=13245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:16.136399292 (13h49m24.074815141s) [upload=13245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:16.136403420 (13h49m24.074819269s) [upload=13245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:16.427080155 (13h49m24.365496004s) [upload=13245]: Done uploading segments... Aug 7 03:58:16.427097879 (13h49m24.365513729s) [upload=13245]: Upload finished. err: <nil> Aug 7 03:58:30.184956648 (13h49m38.123372497s) [upload=13250]: Starting upload Aug 7 03:58:30.185004009 (13h49m38.123419848s) [upload=13250]: Done starting upload Aug 7 03:58:30.185205905 (13h49m38.123621744s) [upload=13250]: Uploading segments... Aug 7 03:58:30.185236314 (13h49m38.123652163s) [upload=13250]: Got next segment. Inline: false Aug 7 03:58:30.185257183 (13h49m38.123673022s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 03:58:30.185260930 (13h49m38.123676779s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 03:58:30.454001938 (13h49m38.392417787s) [seg_pos={0 0}, upload=13250]: Begin upload segment... Aug 7 03:58:30.454044699 (13h49m38.392460548s) [seg_pos={0 0}, upload=13250]: Done begin upload segment. Aug 7 03:58:30.454066982 (13h49m38.392482831s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454100586 (13h49m38.392516435s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454427672 (13h49m38.392843511s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454588760 (13h49m38.393004609s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454692479 (13h49m38.393108328s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454908060 (13h49m38.393323909s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454963637 (13h49m38.393379486s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455232281 (13h49m38.39364813s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455255596 (13h49m38.393671445s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455479334 (13h49m38.393895183s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455506566 (13h49m38.393922405s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455787894 (13h49m38.394203743s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455835545 (13h49m38.394251384s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456019998 (13h49m38.394435848s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456223638 (13h49m38.394639487s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.457172833 (13h49m38.395588682s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:32.206529945 (13h49m40.144945785s) [upload=13250]: Next returned nil segment Aug 7 03:58:32.206545354 (13h49m40.144961203s) [upload=13250]: Waiting for error group managing segments... Aug 7 03:58:32.482974069 (13h49m40.421389908s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.546641532 (13h49m40.485057371s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.547859872 (13h49m40.486275711s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.555284067 (13h49m40.493699906s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.561331598 (13h49m40.499747447s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.562139544 (13h49m40.500555393s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.566721550 (13h49m40.505137399s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.567470363 (13h49m40.505886212s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.576776648 (13h49m40.515192497s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.580990250 (13h49m40.519406099s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.584837120 (13h49m40.523252959s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.585356173 (13h49m40.523772042s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601057214 (13h49m40.539473053s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601086330 (13h49m40.539502219s) [seg_pos={0 0}, upload=13250]: Segment reached optimal threshold of 13 pieces. Aug 7 03:58:32.601212661 (13h49m40.53962851s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601285240 (13h49m40.539701089s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601415780 (13h49m40.539831629s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601463891 (13h49m40.53987973s) [seg_pos={0 0}, upload=13250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:32.601549495 (13h49m40.539965334s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:32.601553784 (13h49m40.539969633s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:32.906871209 (13h49m40.845287058s) [upload=13250]: Done uploading segments... Aug 7 03:58:32.906889855 (13h49m40.845305704s) [upload=13250]: Upload finished. err: <nil> Duration: 5.91743217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.712361452s FAIL (2025-08-07 04:38:00 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.712386831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.061925839s FAIL (2025-08-07 04:37:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== WatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136236350 (13h49m24.074652199s) [noise=true , address=77.90.31.10:4519 , piece_id=XI4WMZRV2Y4HJ2FEJN6G6ZQSIXKPUI2ZSZQAZRW4M4XRTMNGIHPQ, seg_pos={0 0}, upload=13245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136320912 (13h49m24.074736771s) [seg_pos={0 0}, upload=13245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:16.136399292 (13h49m24.074815141s) [upload=13245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:16.136403420 (13h49m24.074819269s) [upload=13245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:16.427080155 (13h49m24.365496004s) [upload=13245]: Done uploading segments... Aug 7 03:58:16.427097879 (13h49m24.365513729s) [upload=13245]: Upload finished. err: <nil> Aug 7 03:58:30.184956648 (13h49m38.123372497s) [upload=13250]: Starting upload Aug 7 03:58:30.185004009 (13h49m38.123419848s) [upload=13250]: Done starting upload Aug 7 03:58:30.185205905 (13h49m38.123621744s) [upload=13250]: Uploading segments... Aug 7 03:58:30.185236314 (13h49m38.123652163s) [upload=13250]: Got next segment. Inline: false Aug 7 03:58:30.185257183 (13h49m38.123673022s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 03:58:30.185260930 (13h49m38.123676779s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 03:58:30.454001938 (13h49m38.392417787s) [seg_pos={0 0}, upload=13250]: Begin upload segment... Aug 7 03:58:30.454044699 (13h49m38.392460548s) [seg_pos={0 0}, upload=13250]: Done begin upload segment. Aug 7 03:58:30.454066982 (13h49m38.392482831s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454100586 (13h49m38.392516435s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454427672 (13h49m38.392843511s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454588760 (13h49m38.393004609s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454692479 (13h49m38.393108328s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454908060 (13h49m38.393323909s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454963637 (13h49m38.393379486s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455232281 (13h49m38.39364813s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455255596 (13h49m38.393671445s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455479334 (13h49m38.393895183s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455506566 (13h49m38.393922405s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455787894 (13h49m38.394203743s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455835545 (13h49m38.394251384s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456019998 (13h49m38.394435848s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456223638 (13h49m38.394639487s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.457172833 (13h49m38.395588682s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:32.206529945 (13h49m40.144945785s) [upload=13250]: Next returned nil segment Aug 7 03:58:32.206545354 (13h49m40.144961203s) [upload=13250]: Waiting for error group managing segments... Aug 7 03:58:32.482974069 (13h49m40.421389908s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.546641532 (13h49m40.485057371s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.547859872 (13h49m40.486275711s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.555284067 (13h49m40.493699906s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.561331598 (13h49m40.499747447s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.562139544 (13h49m40.500555393s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.566721550 (13h49m40.505137399s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.567470363 (13h49m40.505886212s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.576776648 (13h49m40.515192497s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.580990250 (13h49m40.519406099s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.584837120 (13h49m40.523252959s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.585356173 (13h49m40.523772042s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601057214 (13h49m40.539473053s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601086330 (13h49m40.539502219s) [seg_pos={0 0}, upload=13250]: Segment reached optimal threshold of 13 pieces. Aug 7 03:58:32.601212661 (13h49m40.53962851s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601285240 (13h49m40.539701089s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601415780 (13h49m40.539831629s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601463891 (13h49m40.53987973s) [seg_pos={0 0}, upload=13250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:32.601549495 (13h49m40.539965334s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:32.601553784 (13h49m40.539969633s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:32.906871209 (13h49m40.845287058s) [upload=13250]: Done uploading segments... Aug 7 03:58:32.906889855 (13h49m40.845305704s) [upload=13250]: Upload finished. err: <nil> Duration: 6.061951257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.610214891s FAIL (2025-08-07 04:37:33 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.610243556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...............................X......] duration: 1.869397202s FAIL (2025-08-07 04:37:09 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.869419034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.711417678s FAIL (2025-08-07 04:36:55 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.71144522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.190064998s FAIL (2025-08-07 04:36:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== WatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136236350 (13h49m24.074652199s) [noise=true , address=77.90.31.10:4519 , piece_id=XI4WMZRV2Y4HJ2FEJN6G6ZQSIXKPUI2ZSZQAZRW4M4XRTMNGIHPQ, seg_pos={0 0}, upload=13245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136320912 (13h49m24.074736771s) [seg_pos={0 0}, upload=13245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:16.136399292 (13h49m24.074815141s) [upload=13245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:16.136403420 (13h49m24.074819269s) [upload=13245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:16.427080155 (13h49m24.365496004s) [upload=13245]: Done uploading segments... Aug 7 03:58:16.427097879 (13h49m24.365513729s) [upload=13245]: Upload finished. err: <nil> Aug 7 03:58:30.184956648 (13h49m38.123372497s) [upload=13250]: Starting upload Aug 7 03:58:30.185004009 (13h49m38.123419848s) [upload=13250]: Done starting upload Aug 7 03:58:30.185205905 (13h49m38.123621744s) [upload=13250]: Uploading segments... Aug 7 03:58:30.185236314 (13h49m38.123652163s) [upload=13250]: Got next segment. Inline: false Aug 7 03:58:30.185257183 (13h49m38.123673022s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 03:58:30.185260930 (13h49m38.123676779s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 03:58:30.454001938 (13h49m38.392417787s) [seg_pos={0 0}, upload=13250]: Begin upload segment... Aug 7 03:58:30.454044699 (13h49m38.392460548s) [seg_pos={0 0}, upload=13250]: Done begin upload segment. Aug 7 03:58:30.454066982 (13h49m38.392482831s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454100586 (13h49m38.392516435s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454427672 (13h49m38.392843511s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454588760 (13h49m38.393004609s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454692479 (13h49m38.393108328s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454908060 (13h49m38.393323909s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454963637 (13h49m38.393379486s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455232281 (13h49m38.39364813s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455255596 (13h49m38.393671445s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455479334 (13h49m38.393895183s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455506566 (13h49m38.393922405s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455787894 (13h49m38.394203743s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455835545 (13h49m38.394251384s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456019998 (13h49m38.394435848s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456223638 (13h49m38.394639487s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.457172833 (13h49m38.395588682s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:32.206529945 (13h49m40.144945785s) [upload=13250]: Next returned nil segment Aug 7 03:58:32.206545354 (13h49m40.144961203s) [upload=13250]: Waiting for error group managing segments... Aug 7 03:58:32.482974069 (13h49m40.421389908s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.546641532 (13h49m40.485057371s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.547859872 (13h49m40.486275711s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.555284067 (13h49m40.493699906s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.561331598 (13h49m40.499747447s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.562139544 (13h49m40.500555393s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.566721550 (13h49m40.505137399s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.567470363 (13h49m40.505886212s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.576776648 (13h49m40.515192497s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.580990250 (13h49m40.519406099s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.584837120 (13h49m40.523252959s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.585356173 (13h49m40.523772042s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601057214 (13h49m40.539473053s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601086330 (13h49m40.539502219s) [seg_pos={0 0}, upload=13250]: Segment reached optimal threshold of 13 pieces. Aug 7 03:58:32.601212661 (13h49m40.53962851s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601285240 (13h49m40.539701089s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601415780 (13h49m40.539831629s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601463891 (13h49m40.53987973s) [seg_pos={0 0}, upload=13250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:32.601549495 (13h49m40.539965334s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:32.601553784 (13h49m40.539969633s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:32.906871209 (13h49m40.845287058s) [upload=13250]: Done uploading segments... Aug 7 03:58:32.906889855 (13h49m40.845305704s) [upload=13250]: Upload finished. err: <nil> Duration: 6.190125944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.070363508s FAIL (2025-08-07 04:36:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== WatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136236350 (13h49m24.074652199s) [noise=true , address=77.90.31.10:4519 , piece_id=XI4WMZRV2Y4HJ2FEJN6G6ZQSIXKPUI2ZSZQAZRW4M4XRTMNGIHPQ, seg_pos={0 0}, upload=13245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136320912 (13h49m24.074736771s) [seg_pos={0 0}, upload=13245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:16.136399292 (13h49m24.074815141s) [upload=13245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:16.136403420 (13h49m24.074819269s) [upload=13245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:16.427080155 (13h49m24.365496004s) [upload=13245]: Done uploading segments... Aug 7 03:58:16.427097879 (13h49m24.365513729s) [upload=13245]: Upload finished. err: <nil> Aug 7 03:58:30.184956648 (13h49m38.123372497s) [upload=13250]: Starting upload Aug 7 03:58:30.185004009 (13h49m38.123419848s) [upload=13250]: Done starting upload Aug 7 03:58:30.185205905 (13h49m38.123621744s) [upload=13250]: Uploading segments... Aug 7 03:58:30.185236314 (13h49m38.123652163s) [upload=13250]: Got next segment. Inline: false Aug 7 03:58:30.185257183 (13h49m38.123673022s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 03:58:30.185260930 (13h49m38.123676779s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 03:58:30.454001938 (13h49m38.392417787s) [seg_pos={0 0}, upload=13250]: Begin upload segment... Aug 7 03:58:30.454044699 (13h49m38.392460548s) [seg_pos={0 0}, upload=13250]: Done begin upload segment. Aug 7 03:58:30.454066982 (13h49m38.392482831s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454100586 (13h49m38.392516435s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454427672 (13h49m38.392843511s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454588760 (13h49m38.393004609s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454692479 (13h49m38.393108328s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454908060 (13h49m38.393323909s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454963637 (13h49m38.393379486s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455232281 (13h49m38.39364813s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455255596 (13h49m38.393671445s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455479334 (13h49m38.393895183s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455506566 (13h49m38.393922405s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455787894 (13h49m38.394203743s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455835545 (13h49m38.394251384s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456019998 (13h49m38.394435848s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456223638 (13h49m38.394639487s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.457172833 (13h49m38.395588682s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:32.206529945 (13h49m40.144945785s) [upload=13250]: Next returned nil segment Aug 7 03:58:32.206545354 (13h49m40.144961203s) [upload=13250]: Waiting for error group managing segments... Aug 7 03:58:32.482974069 (13h49m40.421389908s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.546641532 (13h49m40.485057371s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.547859872 (13h49m40.486275711s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.555284067 (13h49m40.493699906s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.561331598 (13h49m40.499747447s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.562139544 (13h49m40.500555393s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.566721550 (13h49m40.505137399s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.567470363 (13h49m40.505886212s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.576776648 (13h49m40.515192497s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.580990250 (13h49m40.519406099s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.584837120 (13h49m40.523252959s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.585356173 (13h49m40.523772042s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601057214 (13h49m40.539473053s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601086330 (13h49m40.539502219s) [seg_pos={0 0}, upload=13250]: Segment reached optimal threshold of 13 pieces. Aug 7 03:58:32.601212661 (13h49m40.53962851s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601285240 (13h49m40.539701089s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601415780 (13h49m40.539831629s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601463891 (13h49m40.53987973s) [seg_pos={0 0}, upload=13250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:32.601549495 (13h49m40.539965334s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:32.601553784 (13h49m40.539969633s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:32.906871209 (13h49m40.845287058s) [upload=13250]: Done uploading segments... Aug 7 03:58:32.906889855 (13h49m40.845305704s) [upload=13250]: Upload finished. err: <nil> Duration: 6.070388917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X...............................X.....] duration: 2.166984698s FAIL (2025-08-07 04:36:08 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 2.16701184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.719462047s FAIL (2025-08-07 04:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== WatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136236350 (13h49m24.074652199s) [noise=true , address=77.90.31.10:4519 , piece_id=XI4WMZRV2Y4HJ2FEJN6G6ZQSIXKPUI2ZSZQAZRW4M4XRTMNGIHPQ, seg_pos={0 0}, upload=13245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136320912 (13h49m24.074736771s) [seg_pos={0 0}, upload=13245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:16.136399292 (13h49m24.074815141s) [upload=13245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:16.136403420 (13h49m24.074819269s) [upload=13245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:16.427080155 (13h49m24.365496004s) [upload=13245]: Done uploading segments... Aug 7 03:58:16.427097879 (13h49m24.365513729s) [upload=13245]: Upload finished. err: <nil> Aug 7 03:58:30.184956648 (13h49m38.123372497s) [upload=13250]: Starting upload Aug 7 03:58:30.185004009 (13h49m38.123419848s) [upload=13250]: Done starting upload Aug 7 03:58:30.185205905 (13h49m38.123621744s) [upload=13250]: Uploading segments... Aug 7 03:58:30.185236314 (13h49m38.123652163s) [upload=13250]: Got next segment. Inline: false Aug 7 03:58:30.185257183 (13h49m38.123673022s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 03:58:30.185260930 (13h49m38.123676779s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 03:58:30.454001938 (13h49m38.392417787s) [seg_pos={0 0}, upload=13250]: Begin upload segment... Aug 7 03:58:30.454044699 (13h49m38.392460548s) [seg_pos={0 0}, upload=13250]: Done begin upload segment. Aug 7 03:58:30.454066982 (13h49m38.392482831s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454100586 (13h49m38.392516435s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454427672 (13h49m38.392843511s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454588760 (13h49m38.393004609s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454692479 (13h49m38.393108328s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454908060 (13h49m38.393323909s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454963637 (13h49m38.393379486s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455232281 (13h49m38.39364813s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455255596 (13h49m38.393671445s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455479334 (13h49m38.393895183s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455506566 (13h49m38.393922405s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455787894 (13h49m38.394203743s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455835545 (13h49m38.394251384s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456019998 (13h49m38.394435848s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456223638 (13h49m38.394639487s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.457172833 (13h49m38.395588682s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:32.206529945 (13h49m40.144945785s) [upload=13250]: Next returned nil segment Aug 7 03:58:32.206545354 (13h49m40.144961203s) [upload=13250]: Waiting for error group managing segments... Aug 7 03:58:32.482974069 (13h49m40.421389908s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.546641532 (13h49m40.485057371s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.547859872 (13h49m40.486275711s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.555284067 (13h49m40.493699906s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.561331598 (13h49m40.499747447s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.562139544 (13h49m40.500555393s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.566721550 (13h49m40.505137399s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.567470363 (13h49m40.505886212s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.576776648 (13h49m40.515192497s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.580990250 (13h49m40.519406099s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.584837120 (13h49m40.523252959s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.585356173 (13h49m40.523772042s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601057214 (13h49m40.539473053s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601086330 (13h49m40.539502219s) [seg_pos={0 0}, upload=13250]: Segment reached optimal threshold of 13 pieces. Aug 7 03:58:32.601212661 (13h49m40.53962851s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601285240 (13h49m40.539701089s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601415780 (13h49m40.539831629s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601463891 (13h49m40.53987973s) [seg_pos={0 0}, upload=13250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:32.601549495 (13h49m40.539965334s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:32.601553784 (13h49m40.539969633s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:32.906871209 (13h49m40.845287058s) [upload=13250]: Done uploading segments... Aug 7 03:58:32.906889855 (13h49m40.845305704s) [upload=13250]: Upload finished. err: <nil> Duration: 5.719492906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.607601238s FAIL (2025-08-07 04:35:51 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.60762897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.953964812s FAIL (2025-08-07 04:35:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== WatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136236350 (13h49m24.074652199s) [noise=true , address=77.90.31.10:4519 , piece_id=XI4WMZRV2Y4HJ2FEJN6G6ZQSIXKPUI2ZSZQAZRW4M4XRTMNGIHPQ, seg_pos={0 0}, upload=13245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:16.136320912 (13h49m24.074736771s) [seg_pos={0 0}, upload=13245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:16.136399292 (13h49m24.074815141s) [upload=13245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:16.136403420 (13h49m24.074819269s) [upload=13245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:16.427080155 (13h49m24.365496004s) [upload=13245]: Done uploading segments... Aug 7 03:58:16.427097879 (13h49m24.365513729s) [upload=13245]: Upload finished. err: <nil> Aug 7 03:58:30.184956648 (13h49m38.123372497s) [upload=13250]: Starting upload Aug 7 03:58:30.185004009 (13h49m38.123419848s) [upload=13250]: Done starting upload Aug 7 03:58:30.185205905 (13h49m38.123621744s) [upload=13250]: Uploading segments... Aug 7 03:58:30.185236314 (13h49m38.123652163s) [upload=13250]: Got next segment. Inline: false Aug 7 03:58:30.185257183 (13h49m38.123673022s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 03:58:30.185260930 (13h49m38.123676779s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 03:58:30.454001938 (13h49m38.392417787s) [seg_pos={0 0}, upload=13250]: Begin upload segment... Aug 7 03:58:30.454044699 (13h49m38.392460548s) [seg_pos={0 0}, upload=13250]: Done begin upload segment. Aug 7 03:58:30.454066982 (13h49m38.392482831s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454100586 (13h49m38.392516435s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454427672 (13h49m38.392843511s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454588760 (13h49m38.393004609s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454692479 (13h49m38.393108328s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454908060 (13h49m38.393323909s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.454963637 (13h49m38.393379486s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455232281 (13h49m38.39364813s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455255596 (13h49m38.393671445s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455479334 (13h49m38.393895183s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455506566 (13h49m38.393922405s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455787894 (13h49m38.394203743s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.455835545 (13h49m38.394251384s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456019998 (13h49m38.394435848s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.456223638 (13h49m38.394639487s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:30.457172833 (13h49m38.395588682s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Uploading piece... Aug 7 03:58:32.206529945 (13h49m40.144945785s) [upload=13250]: Next returned nil segment Aug 7 03:58:32.206545354 (13h49m40.144961203s) [upload=13250]: Waiting for error group managing segments... Aug 7 03:58:32.482974069 (13h49m40.421389908s) [noise=true , address=45.91.116.13:4154 , piece_id=VUW4IGZYBTU747MSA53EEMTQSWXJLJABZWTF6KM2DEUQ6C4ZU3MA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.546641532 (13h49m40.485057371s) [noise=true , address=45.91.116.13:4155 , piece_id=77RZ5H3YTCF5YNRDGTIZEQNZ56RAOD2M7TNT7LAEOO3WIYZZLBIA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.547859872 (13h49m40.486275711s) [noise=true , address=45.91.116.14:4191 , piece_id=XN2CJFUBQOFX6KBUQGXLTGWO5AMPSJJCDYF6ITBGKFDWBMHO2HTQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.555284067 (13h49m40.493699906s) [noise=true , address=77.90.31.18:4806 , piece_id=CYXMDPQR5VYF3FKEINVYRQK6OJPMIBTE5WRYU7GN5DPDD7YIRCRA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.561331598 (13h49m40.499747447s) [noise=true , address=45.91.116.6:4973 , piece_id=QB52CSFAPSCQS2HIRVP5QQQJDQAAOMEIEFM537ZYL7BLIOC4LXQA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.562139544 (13h49m40.500555393s) [noise=true , address=45.91.116.11:4119 , piece_id=YRWRT5QTBJL23QFZ2KLJD6TFGZTFE3746CIQUGFS6RNGOXFDEH2A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.566721550 (13h49m40.505137399s) [noise=true , address=45.91.116.7:4010 , piece_id=WLWBH3RUDCH27SNZS3LU3DCF3TI7M66JXHLUBBBXIFFH5NIF63HA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.567470363 (13h49m40.505886212s) [noise=true , address=45.91.116.10:4083 , piece_id=SHSBQD7DFD2G67W3VA6EQSN7ZKPRF7VFDUK22UJ2KFV6TWK56GBQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.576776648 (13h49m40.515192497s) [noise=true , address=77.90.31.10:4517 , piece_id=P7G4OODCQPYGYUM5PA5LPCASJ45NHORQ35WXZ5PWIIANSI4A52OQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.580990250 (13h49m40.519406099s) [noise=true , address=77.90.31.17:4772 , piece_id=FG27XTGPGLPPUVHTJYZVPF3XHEH4G6QGLKXAQY4BV7MJSE7CSOSA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.584837120 (13h49m40.523252959s) [noise=true , address=104.36.82.7:4909 , piece_id=4CZDSCQSBXRRQBWCEOUWJT2C6TLVWY2I6IVERDWAVWZOTEZN6Z7A, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.585356173 (13h49m40.523772042s) [noise=true , address=104.36.82.16:4293 , piece_id=SREUVOSA7477XN3VHUF24OFQLEERODGS63Q7MCFFVUQSLLHZCMRQ, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601057214 (13h49m40.539473053s) [noise=true , address=104.36.82.11:4115 , piece_id=4AQY54Z4E7JIOORSCUPMPODOWWDFR5IYC2WFV3B3K3T3XD3RQVBA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: <nil> Aug 7 03:58:32.601086330 (13h49m40.539502219s) [seg_pos={0 0}, upload=13250]: Segment reached optimal threshold of 13 pieces. Aug 7 03:58:32.601212661 (13h49m40.53962851s) [noise=true , address=45.91.116.8:4045 , piece_id=WGQLWXHFCF4ONW7ZDUGERFZJHUWVW2MZC77DDVCATTD6RYEHRDOA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601285240 (13h49m40.539701089s) [noise=true , address=45.91.116.12:4227 , piece_id=4H7TOOYKHVZ5RRAVU5NDNOTO7NIIBUAQAO4ZOC7JZL5LVAYLZQPA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601415780 (13h49m40.539831629s) [noise=true , address=45.91.116.6:4974 , piece_id=BNJAVQ7IJHPQMRP4L2RKMFS44ROE3ERZXXCGKVMZREW2X5MDH3RA, seg_pos={0 0}, upload=13250]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Aug 7 03:58:32.601463891 (13h49m40.53987973s) [seg_pos={0 0}, upload=13250]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 03:58:32.601549495 (13h49m40.539965334s) [upload=13250]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 03:58:32.601553784 (13h49m40.539969633s) [upload=13250]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 03:58:32.906871209 (13h49m40.845287058s) [upload=13250]: Done uploading segments... Aug 7 03:58:32.906889855 (13h49m40.845305704s) [upload=13250]: Upload finished. err: <nil> Duration: 5.953988879s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...............................X....] duration: 1.839911689s FAIL (2025-08-07 04:35:10 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 11c322f7b1eb2f7e8292e4637fb6451967e8193eeab2f4a37db3ef25b5465093 != before 91e3712b8bb140f6dc7e3b050637abeb30e614ca7e6c42795cd061535652c0ed Duration: 1.839940284s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.611668227s FAIL (2025-08-07 04:35:08 UTC) orbiter warning: operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 46511e3c8ee836342e9b83791f91af91e2d8539c208f329a9198c649a380e28a != before 24e2038caf7e3c9b8dd545c8de8c08ee986034014828d55b4a377a1abf5527b1 Duration: 5.611693695s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)