Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......................................X....] duration: 5.572976267s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:02:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.573006003s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X...................X...............X....X..................] duration: 3.335875351s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:02:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.335902834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.......................................X...] duration: 5.71643938s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:01:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.716471712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X..........X.........X................X...X...X...........] duration: 5.648531813s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:01:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.648558242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.951431714s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 07:01:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.951460279s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...................................X....] duration: 5.804471848s trace: https://jaeger.dev.storj.io/trace/270563580d82773d FAIL (2025-12-09 07:01:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.804500282s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.461182079s trace: https://jaeger.dev.storj.io/trace/66b2ccb4b553cce8 FAIL (2025-12-09 07:01:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.461212176s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 4.949526417s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:01:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.94955895s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X..........X.........X................X...X...X..........] duration: 5.784051036s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:01:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.784096221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.......................................X..] duration: 5.576551348s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:01:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.576579951s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X...................X...............X....X.................] duration: 3.533800357s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:01:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53382757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.997313135s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 07:00:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.997352048s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...................................X...] duration: 5.661622186s trace: https://jaeger.dev.storj.io/trace/270563580d82773d FAIL (2025-12-09 07:00:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.661654048s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.514882467s trace: https://jaeger.dev.storj.io/trace/66b2ccb4b553cce8 FAIL (2025-12-09 07:00:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.514984641s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X..........X.........X................X...X...X.........] duration: 5.969092149s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:00:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.96916603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.......................................X.] duration: 5.704684483s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:00:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.704702598s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 4.895804844s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:00:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.895838429s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.428953133s trace: https://jaeger.dev.storj.io/trace/66b2ccb4b553cce8 FAIL (2025-12-09 07:00:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.428990515s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.......................................X] duration: 4m33.544011524s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:00:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 4m33.54403572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.949109083s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 07:00:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.949221225s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X...................X...............X....X................] duration: 3.448250025s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 06:59:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44827901s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....X..........X.........X................X...X...X........] duration: 5.762359865s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 06:59:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.762422034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 10.895565177s trace: https://jaeger.dev.storj.io/trace/66b2ccb4b553cce8 FAIL (2025-12-09 06:59:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 10.895581708s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 4.975959107s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 06:59:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.975976459s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...................................X..] duration: 5.645847761s trace: https://jaeger.dev.storj.io/trace/270563580d82773d FAIL (2025-12-09 06:59:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.645875434s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.935479412s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 06:59:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.935502274s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...................................X.] duration: 5.778205746s trace: https://jaeger.dev.storj.io/trace/270563580d82773d FAIL (2025-12-09 06:59:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.778235382s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X...................X...............X....X...............] duration: 3.479789779s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 06:59:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479813315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.901797039s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 06:59:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.901826724s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...................................X] duration: 31.20312933s trace: https://jaeger.dev.storj.io/trace/270563580d82773d FAIL (2025-12-09 06:58:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 31.203155148s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.399193241s trace: https://jaeger.dev.storj.io/trace/66b2ccb4b553cce8 FAIL (2025-12-09 06:58:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.399235802s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 4.829200429s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 06:58:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.829265043s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......X..........X.........X................X...X...X.......] duration: 6.502778621s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 06:58:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.502805322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.93113592s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 06:58:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.931165956s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.051365803s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 06:58:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.051393598s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.444975881s trace: https://jaeger.dev.storj.io/trace/66b2ccb4b553cce8 FAIL (2025-12-09 06:58:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.445005497s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X...................X...............X....X..............] duration: 3.526635896s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 06:58:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526664761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.94367413s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 06:57:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.943702504s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......X..........X.........X................X...X...X......] duration: 5.872755521s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 06:57:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.872788285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.966886981s trace: https://jaeger.dev.storj.io/trace/29c19112d1fd5982 FAIL (2025-12-09 06:57:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.966925263s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 4.951666601s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 06:57:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.951711246s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.111694842s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 06:57:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.111732915s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.36172492s trace: https://jaeger.dev.storj.io/trace/66b2ccb4b553cce8 FAIL (2025-12-09 06:57:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.361755329s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X...................X...............X....X.............] duration: 3.448316545s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 06:57:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448371138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........X..........X.........X................X...X...X.....] duration: 5.918228215s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 06:57:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.918256869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.967874474s trace: https://jaeger.dev.storj.io/trace/29c19112d1fd5982 FAIL (2025-12-09 06:57:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.967904122s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.981834774s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 06:56:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.981861625s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.920944066s trace: https://jaeger.dev.storj.io/trace/61580621ad164dbd FAIL (2025-12-09 06:56:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.920971007s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.433437915s trace: https://jaeger.dev.storj.io/trace/66b2ccb4b553cce8 FAIL (2025-12-09 06:56:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.433470056s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........X..........X.........X................X...X...X....] duration: 5.644355511s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 06:56:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.644381719s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)