Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X....XX..] duration: 7.321231961s FAIL (2025-08-19 16:51:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.321258864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.247097743s FAIL (2025-08-19 16:50:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.247118044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 6.692581558s FAIL (2025-08-19 16:50:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.692602349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X....X..] duration: 6.056407102s FAIL (2025-08-19 16:50:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.056427651s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X.X...X...X..............X......X.XX..] duration: 5.843839089s FAIL (2025-08-19 16:50:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.843863815s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X...............X........................] duration: 2.417095678s FAIL (2025-08-19 16:50:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.417116638s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 6.870456186s FAIL (2025-08-19 16:50:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.870477598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X....XX.] duration: 5.970639349s FAIL (2025-08-19 16:50:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.97066589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 11.239613851s FAIL (2025-08-19 16:50:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 11.239640793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X.X...X...X..............X......X.XX.] duration: 13.252028414s FAIL (2025-08-19 16:50:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 13.252074141s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X.X...X...X..............X......X.XX] duration: 1m0.273504521s FAIL (2025-08-19 16:49:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 19 16:48:51.452491912 (263h22m45.883572043s) [upload=273834]: Starting upload Aug 19 16:48:51.452531028 (263h22m45.883611159s) [upload=273834]: Done starting upload Aug 19 16:48:51.452781006 (263h22m45.883861137s) [upload=273834]: Uploading segments... Aug 19 16:48:51.452853734 (263h22m45.883933866s) [upload=273834]: Got next segment. Inline: false Aug 19 16:48:51.452892670 (263h22m45.883972871s) [upload=273834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:51.452899813 (263h22m45.883980004s) [upload=273834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:51.962807465 (263h22m46.393887596s) [seg_pos={0 0}, upload=273834]: Begin upload segment... Aug 19 16:48:51.962852621 (263h22m46.393932743s) [seg_pos={0 0}, upload=273834]: Done begin upload segment. Aug 19 16:48:51.962875075 (263h22m46.393955206s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.962917746 (263h22m46.393997877s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963271022 (263h22m46.394351153s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963372556 (263h22m46.394452687s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963580684 (263h22m46.394660815s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963653934 (263h22m46.394734066s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963841874 (263h22m46.394922005s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.963910455 (263h22m46.394990586s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964097293 (263h22m46.395177424s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964163640 (263h22m46.395243771s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964400383 (263h22m46.395480514s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964624241 (263h22m46.395704372s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.964832199 (263h22m46.39591233s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965048673 (263h22m46.396128804s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965259626 (263h22m46.396339747s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965452415 (263h22m46.396532546s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965641637 (263h22m46.396721768s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:51.965869794 (263h22m46.396949925s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Uploading piece... Aug 19 16:48:53.162668249 (263h22m47.59374838s) [upload=273834]: Next returned nil segment Aug 19 16:48:53.162683478 (263h22m47.593763599s) [upload=273834]: Waiting for error group managing segments... Aug 19 16:48:53.488132069 (263h22m47.91921223s) [noise=true , address=66.23.193.175:4556 , piece_id=FBS7KZJF3HULYZZJDNLNOKMR5HSCR224IY7ZU4FUKZC5YMIIOAJA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.887114541 (263h22m48.318194672s) [noise=true , address=79.127.219.51:4211 , piece_id=OUB6QKVWCRJATMQFLAQ7UK3LJ4DILSLRAGFDP5YXROEXHZCPWGZA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.925197211 (263h22m48.356277342s) [noise=true , address=156.146.43.225:4452 , piece_id=3T6HWTUPTX6A7BZULB4RSKGUO6GNDIHQGUP5W3EESJBWZGN7LLEQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.961937701 (263h22m48.393017832s) [noise=true , address=104.36.82.11:4127 , piece_id=6ACOOH24TTEMA4OQP6OFNHIABWDW3NDUVD44FSRIFLPTSVPHXOOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:53.992310049 (263h22m48.42339018s) [noise=true , address=79.127.201.215:4343 , piece_id=JF5HESY3ZARRZWZJHGTIG2BEZWWMXPHNJF5ALABQBAOVGJCHKI5A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.045243820 (263h22m48.476323961s) [noise=true , address=104.36.82.35:4606 , piece_id=R37F22AIPFPV45K2PRWAJNONXQEDBJQPZGQKQ3LSHAQIBPKRXLHA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.059875193 (263h22m48.490955324s) [noise=true , address=104.36.82.21:4478 , piece_id=XJWKXZ45KPRM3QUYXVIYU75M24GOCWGG5E3K4V62225XF3TGSS4Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.095629989 (263h22m48.52671012s) [noise=true , address=104.36.82.23:4470 , piece_id=5PDSSXU4HWGHBN62JJQBKHDQ6YYJVBWZZYSCJGKOPTZAZZUCAB6Q, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.103250568 (263h22m48.534330699s) [noise=true , address=104.36.82.51:4052 , piece_id=7HLVSHJCDKI3TNKFETBUAY25HVIOPKAMSYQ6Q5BJ2NCJTQOTETTA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:48:54.159963800 (263h22m48.591043921s) [noise=true , address=104.36.82.60:4341 , piece_id=JCAHPCHE5IPHX35ZCDQ27I3DZ3R347HFRQWNBCNBXU6P3O7PTONA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:37.214809078 (263h23m31.64588921s) [noise=true , address=77.90.31.17:4793 , piece_id=K3GCAOBIKBJV6Q3KOFXL5SEUFYE46MKJU5IAYNDA25IDCICTOFCQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: <nil> Aug 19 16:49:50.811468753 (263h23m45.242548894s) [noise=true , address=77.90.31.10:4529 , piece_id=HK5DLHGNULVF2RX7NNKFYN5JIFJB3LDEYXLXO3UN4D5OO4DJOW4A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: read tcp 188.166.225.170:57426->77.90.31.10:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811576489 (263h23m45.24265662s) [noise=true , address=45.91.116.14:4209 , piece_id=5KAIVWUEWAG5NW6QGOAJLIMUADHZ6R4YTNAKTTXAGBRUEYXDAE7A, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:12LWbAxsrDcyV31SxJNiVNYs7ERKAf6RTuKpE596UrYJAnXMndR, address:45.91.116.14:4209): protocol: rpc: dial tcp 45.91.116.14:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811688974 (263h23m45.242769105s) [noise=true , address=45.91.116.9:4286 , piece_id=WYJJULBB3QVG2F46C52OCJWAMHVORGI7MDQS3CSYLF7REGAZQ3OQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, address:45.91.116.9:4286): protocol: read tcp 188.166.225.170:40032->45.91.116.9:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811727057 (263h23m45.242807188s) [noise=true , address=45.91.116.11:4130 , piece_id=5IZILPAZ4UKTFE766TWKT672JHDPWIQJI3TMOTASODK7NYG5BKPQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: read tcp 188.166.225.170:46860->45.91.116.11:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811808934 (263h23m45.242889085s) [noise=true , address=45.91.116.7:4020 , piece_id=5NNTOPQ5BC7T2TLQ7NWSO3E6R4NDPB3JPXPMPRGG6I7YJU6QS4CQ, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, address:45.91.116.7:4020): protocol: read tcp 188.166.225.170:48966->45.91.116.7:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811874559 (263h23m45.24295469s) [noise=true , address=77.90.31.18:4828 , piece_id=NFXT5LHIVX4G6DXMB4UWI37XCXLNHBRLVLXSJWTYAYN2JKCHRZQA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: read tcp 188.166.225.170:47612->77.90.31.18:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811914556 (263h23m45.242994687s) [noise=true , address=77.90.31.15:4713 , piece_id=26S5ETN4OVNEL7IYYAIUBLP6UDS2QSSXCHD4H6YNHEQNY6FPOBOA, seg_pos={0 0}, upload=273834]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: read tcp 188.166.225.170:41506->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:50.811942720 (263h23m45.243022841s) [seg_pos={0 0}, upload=273834]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 19 16:49:51.082762534 (263h23m45.513842665s) [upload=273834]: Done uploading segments... Aug 19 16:49:51.082792671 (263h23m45.513872803s) [upload=273834]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.273522496s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 56.895139766s FAIL (2025-08-19 16:49:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 56.895166288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X....XX] duration: 30.230151202s FAIL (2025-08-19 16:49:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 19 16:49:12.801154313 (263h23m7.232234445s) [upload=273839]: Starting upload Aug 19 16:49:12.801215942 (263h23m7.232296073s) [upload=273839]: Done starting upload Aug 19 16:49:12.801330351 (263h23m7.232410482s) [upload=273839]: Uploading segments... Aug 19 16:49:12.801425372 (263h23m7.232505503s) [upload=273839]: Got next segment. Inline: false Aug 19 16:49:12.801455880 (263h23m7.232536012s) [upload=273839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:49:12.801461982 (263h23m7.232542113s) [upload=273839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:49:13.036708228 (263h23m7.467788359s) [seg_pos={0 0}, upload=273839]: Begin upload segment... Aug 19 16:49:13.036749197 (263h23m7.467829328s) [seg_pos={0 0}, upload=273839]: Done begin upload segment. Aug 19 16:49:13.036771971 (263h23m7.467852092s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.036780256 (263h23m7.467860437s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037135917 (263h23m7.468216048s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037326592 (263h23m7.468406723s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037403259 (263h23m7.46848339s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037667814 (263h23m7.468747945s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037761764 (263h23m7.468841896s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.037943331 (263h23m7.469023462s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038070565 (263h23m7.469150696s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038168622 (263h23m7.469248753s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038300104 (263h23m7.469380235s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038396789 (263h23m7.46947692s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038622241 (263h23m7.469702362s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038679350 (263h23m7.469759481s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.038854735 (263h23m7.469934856s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:13.039048045 (263h23m7.470128166s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Uploading piece... Aug 19 16:49:14.749064365 (263h23m9.180144496s) [upload=273839]: Next returned nil segment Aug 19 16:49:14.749117217 (263h23m9.180197348s) [upload=273839]: Waiting for error group managing segments... Aug 19 16:49:15.090317863 (263h23m9.521397994s) [noise=true , address=104.36.82.2:4731 , piece_id=J35JV3B5HGY32B5MI4DW554STFHV6FMICKQI22K4M2TWQAU4LK6Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.132109007 (263h23m9.563189138s) [noise=true , address=104.36.82.2:4730 , piece_id=NCEH7LM5VSSYK5L3WYC3KSZ6LRFX7LBMB6IF7AEEVAG3A56JEX2Q, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.222090156 (263h23m9.653170287s) [noise=true , address=104.36.82.12:4149 , piece_id=JVDLRTR4KG667EVONN6SSLARFPFCEWSV6Q7HEFH7TRW6LTTINZLQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.328041780 (263h23m9.759121911s) [noise=true , address=104.36.82.7:4909 , piece_id=U62EINXBZ5IXLTITKISTEE2KGPFH7VGPLDYGDIALF4Y6QKN4UE4A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:15.353998781 (263h23m9.785078912s) [noise=true , address=104.36.82.8:4971 , piece_id=M6RNUVJBSUO4Z5TMGWIBAFMGJMDKRFJU5BQUFYB4NP33QRFCCF2A, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:36.709543777 (263h23m31.140623898s) [noise=true , address=45.91.116.6:4974 , piece_id=L6M7DWSJGFPBXEZ3PC5UOUZGLRDCKI24L24RMIGYHKHVPK3GJTYQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:37.041062020 (263h23m31.472142141s) [noise=true , address=45.91.116.7:4010 , piece_id=HEEX7W5N5M5K775WJQYEYHIHV3FOBGHFKYEG6M2GKBBU7BFORVZQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.183707791 (263h23m32.614787922s) [noise=true , address=45.91.116.11:4117 , piece_id=EWJWVC6BGPS7OW3TK6JETRUMQ7P3KDT2GOLSJOU5NADIH7CZXWAQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:38.694075084 (263h23m33.125155215s) [noise=true , address=45.91.116.12:4225 , piece_id=GSYBYST7VPELT4GYLSYPD3VQH5OGDUZGVDWSGOJYAXBWMVYJZ2WQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.716788163 (263h23m34.147868294s) [noise=true , address=45.91.116.5:4297 , piece_id=YVSRADMZ6O6SDG4SJQECVGL5MJCM7Q2AVK4FHVF3CFCNYT436ENA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:39.851890717 (263h23m34.282970848s) [noise=true , address=45.91.116.8:4047 , piece_id=XCMLU7NGNXIARKHX4TNJLS6UGJON7HTJRLVS5IQS5ZD2JTQIYFXA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:40.293306092 (263h23m34.724386243s) [noise=true , address=45.91.116.10:4082 , piece_id=S62YL4AXNL6LW3RWFM3CJDGGW75236RMH35PPZWTLC56O5FI5CQQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: <nil> Aug 19 16:49:42.156929229 (263h23m36.58800936s) [noise=true , address=77.90.31.21:4878 , piece_id=2GAMMSPYH5TDIJNR3LYHLNSVYXC4DNH3CGB6QS3XYMGLAQIBNMIQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157075889 (263h23m36.58815601s) [noise=true , address=77.90.31.14:4661 , piece_id=B3RBZRJWNBWVC4UGH666QTS6ZYSR4FRILM63NRNE2ECSIG2X4PTQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:42.157126075 (263h23m36.588206206s) [noise=true , address=77.90.31.12:4592 , piece_id=5TMYE4OAKYO7N4FD2PQESGJRFV4OLBKU3I5RBO4P5FWXOTZCJLJQ, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46438->77.90.31.12:4592: use of closed network connection Aug 19 16:49:42.157170951 (263h23m36.588251082s) [noise=true , address=45.91.116.12:4226 , piece_id=N7DO2PWSFQLQSC2AB7H3DWSX3Z5V5NHQOM3UCHYWAPQJFUXIXZEA, seg_pos={0 0}, upload=273839]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:34242->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:42.157215747 (263h23m36.588295878s) [seg_pos={0 0}, upload=273839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 19 16:49:42.385478364 (263h23m36.816558495s) [upload=273839]: Done uploading segments... Aug 19 16:49:42.385503091 (263h23m36.816583212s) [upload=273839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230172633s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 7.26463528s FAIL (2025-08-19 16:49:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.264649657s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 7.820252283s FAIL (2025-08-19 16:49:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.820264276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 1m0.001244768s FAIL (2025-08-19 16:49:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001268105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.225526327s FAIL (2025-08-19 16:49:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 19 16:48:42.036175830 (263h22m36.467255971s) [upload=273830]: Starting upload Aug 19 16:48:42.036249500 (263h22m36.467329631s) [upload=273830]: Done starting upload Aug 19 16:48:42.036382705 (263h22m36.467462876s) [upload=273830]: Uploading segments... Aug 19 16:48:42.036575073 (263h22m36.467655204s) [upload=273830]: Got next segment. Inline: false Aug 19 16:48:42.036618537 (263h22m36.467698668s) [upload=273830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:48:42.036622684 (263h22m36.467702815s) [upload=273830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:48:42.269318081 (263h22m36.700398202s) [seg_pos={0 0}, upload=273830]: Begin upload segment... Aug 19 16:48:42.269357146 (263h22m36.700437267s) [seg_pos={0 0}, upload=273830]: Done begin upload segment. Aug 19 16:48:42.269376102 (263h22m36.700456233s) [noise=true , address=104.36.82.8:4971 , piece_id=7RUGKEFOT7QM3WHK3NQQN6K2BRHTHBBIDL5QZLNWFKWX4E5JS3PA, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.269456255 (263h22m36.700536446s) [noise=true , address=104.36.82.5:4839 , piece_id=R25OMVJZJPABKPH6NKZGT54M545STYPL3ZCGVRDLVUBBQOSQOEVQ, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.269737473 (263h22m36.700817604s) [noise=true , address=45.91.116.9:4261 , piece_id=EV3BFXSX7G2OWWHCQBMXP45Q3IUL6PXEXQMYR55FFHIFENOCJ4KA, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.270056193 (263h22m36.701136324s) [noise=true , address=77.90.31.15:4698 , piece_id=ORMUK6LVIQ6PDNQETA2MCVVAUDVG5S3DIADV65WYKRJUNQRYKA6Q, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.270323445 (263h22m36.701403576s) [noise=true , address=45.91.116.11:4117 , piece_id=KUELMNUL6LAP5SNUICNES5P65S5RPQ345RPRZSD7I3HY2RWDBVGA, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.270594283 (263h22m36.701674414s) [noise=true , address=45.91.116.4:4940 , piece_id=2DM4KVDDJYS2FKVMOGIQH4BXDPCQB2SKHFYXE6ENI32XPKBFMB5A, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.270612227 (263h22m36.701692358s) [noise=true , address=104.36.82.14:4222 , piece_id=36DILLTIVVFMOEN65DWG5EMEBNB6Z3HRFJZ7YFSFBA6T5WOAKYUA, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.270831917 (263h22m36.701912048s) [noise=true , address=104.36.82.7:4909 , piece_id=D4AKGMX4Y35OLK6YY7NBGT4MKFPT7DXE3TRU7PALSQKWLELGHN3A, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.271068310 (263h22m36.702148441s) [noise=true , address=104.36.82.9:4005 , piece_id=3SMVIZQAU7VRF4JUA3UTRSDBZFX5XEJ3ZD7CIQCUG2SJEJA2JKNQ, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.271257422 (263h22m36.702337553s) [noise=true , address=104.36.82.2:4731 , piece_id=XBC4U5343ZPW3QKCGNV2PQ4VPUDRBT5QG2SLFMJEJYE6WUIWBLUQ, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.271296696 (263h22m36.702376817s) [noise=true , address=45.91.116.4:4937 , piece_id=GXDKB5SLD3BF2CEUEEV72337FZUISA2FT375CXTBMTPV3Q3BGVBQ, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.271519963 (263h22m36.702600094s) [noise=true , address=104.36.82.6:4875 , piece_id=YXWHKSVKMMOVHM2COUNDO3RJK5DGBZNHQASE7E6UTNAWSDMGKVPQ, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.271872368 (263h22m36.702952499s) [noise=true , address=104.36.82.12:4151 , piece_id=TQ2XDIZ4RFPK67PYOCNSICQD6HD4Y6C4XO5Q5NFXLFPRKE3PP6SQ, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.272419004 (263h22m36.703499125s) [noise=true , address=104.36.82.3:4765 , piece_id=YPSIPRGAJJHEEQUEDK763GQMVRVYFP4CWWAAPR5PQUUAJLTT7ZDQ, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.272784914 (263h22m36.703865045s) [noise=true , address=45.91.116.10:4083 , piece_id=ZLSIBFJYXPUFXUDYWMFIWLVNMTNORX2XBRSM4NPNUALS7F6AVJ7A, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:42.273184718 (263h22m36.704264849s) [noise=true , address=104.36.82.11:4114 , piece_id=7SOGD3BYXG3ZU5ZJWWUNKREEGVSLJUQSOERLVJRLGUEAJGQNNI3Q, seg_pos={0 0}, upload=273830]: Uploading piece... Aug 19 16:48:44.018977906 (263h22m38.450058027s) [upload=273830]: Next returned nil segment Aug 19 16:48:44.018993396 (263h22m38.450073528s) [upload=273830]: Waiting for error group managing segments... Aug 19 16:48:44.326769496 (263h22m38.757849627s) [noise=true , address=104.36.82.12:4151 , piece_id=TQ2XDIZ4RFPK67PYOCNSICQD6HD4Y6C4XO5Q5NFXLFPRKE3PP6SQ, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.339565566 (263h22m38.770645698s) [noise=true , address=104.36.82.3:4765 , piece_id=YPSIPRGAJJHEEQUEDK763GQMVRVYFP4CWWAAPR5PQUUAJLTT7ZDQ, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.343333175 (263h22m38.774413306s) [noise=true , address=104.36.82.11:4114 , piece_id=7SOGD3BYXG3ZU5ZJWWUNKREEGVSLJUQSOERLVJRLGUEAJGQNNI3Q, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.368085200 (263h22m38.799165331s) [noise=true , address=104.36.82.7:4909 , piece_id=D4AKGMX4Y35OLK6YY7NBGT4MKFPT7DXE3TRU7PALSQKWLELGHN3A, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.395127266 (263h22m38.826207397s) [noise=true , address=104.36.82.9:4005 , piece_id=3SMVIZQAU7VRF4JUA3UTRSDBZFX5XEJ3ZD7CIQCUG2SJEJA2JKNQ, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.453337913 (263h22m38.884418044s) [noise=true , address=104.36.82.2:4731 , piece_id=XBC4U5343ZPW3QKCGNV2PQ4VPUDRBT5QG2SLFMJEJYE6WUIWBLUQ, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.454780744 (263h22m38.885860876s) [noise=true , address=104.36.82.6:4875 , piece_id=YXWHKSVKMMOVHM2COUNDO3RJK5DGBZNHQASE7E6UTNAWSDMGKVPQ, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.471425999 (263h22m38.902506121s) [noise=true , address=104.36.82.14:4222 , piece_id=36DILLTIVVFMOEN65DWG5EMEBNB6Z3HRFJZ7YFSFBA6T5WOAKYUA, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.477557059 (263h22m38.90863719s) [noise=true , address=104.36.82.5:4839 , piece_id=R25OMVJZJPABKPH6NKZGT54M545STYPL3ZCGVRDLVUBBQOSQOEVQ, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:48:44.483532060 (263h22m38.914612191s) [noise=true , address=104.36.82.8:4971 , piece_id=7RUGKEFOT7QM3WHK3NQQN6K2BRHTHBBIDL5QZLNWFKWX4E5JS3PA, seg_pos={0 0}, upload=273830]: Done uploading piece. err: <nil> Aug 19 16:49:11.394625577 (263h23m5.825705708s) [noise=true , address=77.90.31.15:4698 , piece_id=ORMUK6LVIQ6PDNQETA2MCVVAUDVG5S3DIADV65WYKRJUNQRYKA6Q, seg_pos={0 0}, upload=273830]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:50826->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:11.394908609 (263h23m5.82598874s) [noise=true , address=45.91.116.10:4083 , piece_id=ZLSIBFJYXPUFXUDYWMFIWLVNMTNORX2XBRSM4NPNUALS7F6AVJ7A, seg_pos={0 0}, upload=273830]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:42890->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:11.394941752 (263h23m5.826021883s) [noise=true , address=45.91.116.11:4117 , piece_id=KUELMNUL6LAP5SNUICNES5P65S5RPQ345RPRZSD7I3HY2RWDBVGA, seg_pos={0 0}, upload=273830]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 188.166.225.170:52894->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:11.394969154 (263h23m5.826049275s) [noise=true , address=45.91.116.4:4937 , piece_id=GXDKB5SLD3BF2CEUEEV72337FZUISA2FT375CXTBMTPV3Q3BGVBQ, seg_pos={0 0}, upload=273830]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:41308->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:11.395011015 (263h23m5.826091146s) [noise=true , address=45.91.116.9:4261 , piece_id=EV3BFXSX7G2OWWHCQBMXP45Q3IUL6PXEXQMYR55FFHIFENOCJ4KA, seg_pos={0 0}, upload=273830]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: read tcp 188.166.225.170:59666->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:11.395038948 (263h23m5.826119079s) [noise=true , address=45.91.116.4:4940 , piece_id=2DM4KVDDJYS2FKVMOGIQH4BXDPCQB2SKHFYXE6ENI32XPKBFMB5A, seg_pos={0 0}, upload=273830]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 188.166.225.170:60706->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:11.395052303 (263h23m5.826132434s) [seg_pos={0 0}, upload=273830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 19 16:49:11.618547508 (263h23m6.04962764s) [upload=273830]: Done uploading segments... Aug 19 16:49:11.618570823 (263h23m6.049650954s) [upload=273830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.225546225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X...............X.......................] duration: 2.312269046s FAIL (2025-08-19 16:49:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.312296628s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.001390696s FAIL (2025-08-19 16:49:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001411335s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.868843838s FAIL (2025-08-19 16:48:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 21338s) [upload=273768]: Uploading segments... Aug 19 16:44:56.941197143 (263h18m51.372277274s) [upload=273768]: Got next segment. Inline: false Aug 19 16:44:56.941232320 (263h18m51.372312451s) [upload=273768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:44:56.941236459 (263h18m51.37231659s) [upload=273768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:44:57.179169145 (263h18m51.610249277s) [seg_pos={0 0}, upload=273768]: Begin upload segment... Aug 19 16:44:57.179239420 (263h18m51.610319551s) [seg_pos={0 0}, upload=273768]: Done begin upload segment. Aug 19 16:44:57.179269538 (263h18m51.610349669s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179307400 (263h18m51.610387551s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179784723 (263h18m51.610864854s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179905404 (263h18m51.610985535s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180182484 (263h18m51.611262616s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180645561 (263h18m51.611725682s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180924785 (263h18m51.612004916s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181130468 (263h18m51.612210599s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181461401 (263h18m51.612541522s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181689417 (263h18m51.612769548s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181988539 (263h18m51.61306867s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182215543 (263h18m51.613295674s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182547829 (263h18m51.61362796s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182768241 (263h18m51.613848372s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182984705 (263h18m51.614064836s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.183204966 (263h18m51.614285098s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:58.961741539 (263h18m53.39282167s) [upload=273768]: Next returned nil segment Aug 19 16:44:58.961758922 (263h18m53.392839054s) [upload=273768]: Waiting for error group managing segments... Aug 19 16:44:59.254505408 (263h18m53.685585529s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.280332548 (263h18m53.711412679s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.288599876 (263h18m53.719680007s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.317155488 (263h18m53.748235619s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.331253149 (263h18m53.76233328s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.346011735 (263h18m53.777091866s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.353737125 (263h18m53.784817246s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.363453003 (263h18m53.794533134s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:45:26.286817635 (263h19m20.717897766s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:40788->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286870046 (263h19m20.717950177s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:57632->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286906706 (263h19m20.717986837s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:55282->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286981619 (263h19m20.71806175s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; context deadline exceeded Aug 19 16:45:26.287080398 (263h19m20.718160529s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: rpc: dial tcp 45.91.116.8:4048: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287106397 (263h19m20.718186528s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:43594->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287139521 (263h19m20.718219652s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:45424->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287163056 (263h19m20.718243187s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:59538->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287183124 (263h19m20.718263255s) [seg_pos={0 0}, upload=273768]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:26.515324579 (263h19m20.94640471s) [upload=273768]: Done uploading segments... Aug 19 16:45:26.515351451 (263h19m20.946431582s) [upload=273768]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.868887091s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.183798541s FAIL (2025-08-19 16:48:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.183818479s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...............X......................] duration: 1.87081011s FAIL (2025-08-19 16:48:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 1.870835678s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.706750982s FAIL (2025-08-19 16:48:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.706774107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.711127568s FAIL (2025-08-19 16:48:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.711158727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X.X...X...X..............X......X.X] duration: 8.689482393s FAIL (2025-08-19 16:47:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 5e1982a2d6896698cc5a4622395980a60a9c1562017183f5fdcf7d3cb8f559c4 != before a374966880ccd60f338788c55184f66df0c8a658fe79f19a9fe1641ac40cdfd1 ======LOGS====== Aug 19 16:47:36.736941377 (263h21m31.168021498s) [upload=273812]: Starting upload Aug 19 16:47:36.736990171 (263h21m31.168070292s) [upload=273812]: Done starting upload Aug 19 16:47:36.737091314 (263h21m31.168171445s) [upload=273812]: Uploading segments... Aug 19 16:47:36.737173101 (263h21m31.168253232s) [upload=273812]: Got next segment. Inline: false Aug 19 16:47:36.737197136 (263h21m31.168277267s) [upload=273812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:47:36.737201034 (263h21m31.168281165s) [upload=273812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:47:37.200389505 (263h21m31.631469737s) [seg_pos={0 0}, upload=273812]: Begin upload segment... Aug 19 16:47:37.200442817 (263h21m31.631522949s) [seg_pos={0 0}, upload=273812]: Done begin upload segment. Aug 19 16:47:37.200457665 (263h21m31.631537786s) [noise=true , address=104.36.82.44:4873 , piece_id=2CAUKQ6RUCMWLY4YNDWA5LZWVFIDHIHJQH3DLNIQC4RIPFFK7Y5Q, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.200465370 (263h21m31.631545501s) [noise=true , address=104.36.82.49:4008 , piece_id=7737UXHHDA2QNVTRMBUJWPMGCZ5CCLSQ34YIQK6ZS3L4CUMXGGPA, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.200863763 (263h21m31.631943884s) [noise=true , address=77.90.31.12:4623 , piece_id=HO6MWVF2QZ3OF6SY2UZ6ZHYUCOK6KWYFZDOV5A3YZVROU3JROEPA, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.200893419 (263h21m31.63197355s) [noise=true , address=104.36.82.51:4070 , piece_id=Y6MC5XWZBVMBBE44O2IE3FZAGUCMN4GXULPNUN2TVG4EIUQ3JMKQ, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.201151362 (263h21m31.632231483s) [noise=true , address=104.36.82.6:4896 , piece_id=HSQ4SX5X5Q3F4SDXDBTDUR5XMUPZKLMXLHMO3JNCXHJB57L3BB5A, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.201429935 (263h21m31.632510076s) [noise=true , address=104.36.82.18:4376 , piece_id=ST4DA6OBHHBNMIU65QUDEA74KRQJYBVUKOHSMYY5Y6ZS3BAUHDUQ, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.201749116 (263h21m31.632829247s) [noise=true , address=104.36.82.50:4024 , piece_id=ZTLIWXBXFJCQD4X4KDNCOZNWUBMN7CCQSPL36GM7YJOJFOBJ5ZBQ, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.201785285 (263h21m31.632865416s) [noise=true , address=77.90.31.11:4565 , piece_id=ALIH2EDV7RGGNR5ALESOUPXP3BGWTEO55SL4ODHQDIV2CAKQZWJA, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.202044151 (263h21m31.633124282s) [noise=true , address=45.91.116.11:4146 , piece_id=KGOCYLZXVMAKOBNSAVJPGADSBZWCBIVS3UBUDRYP2CYNJGXQF7QA, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.202282907 (263h21m31.633363028s) [noise=true , address=104.36.82.40:4950 , piece_id=BCMHBHHHDOYRWMU2DWMK36PVWWYTPGOBW2LP3TN3V7RS7OBYUH2A, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.202445498 (263h21m31.633525629s) [noise=true , address=45.91.116.8:4078 , piece_id=ONY7KHFCBUAGM7LBYVC4CEKWGYTXPYAZGKITWWQLILU7CTXICOOQ, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.202693763 (263h21m31.633773894s) [noise=true , address=104.36.82.60:4349 , piece_id=KH2AJTUXDYN5JDJJ3KB2MZUFC5BSFZ5IMRLCC2FVO3FYVPVJXFFQ, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.202915508 (263h21m31.633995639s) [noise=true , address=77.90.31.17:4786 , piece_id=WXHOQG2NSD5Z53U4MH7FUAXPBTUQBP7JWH7GKZBNZZMHZHV6VLBQ, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.203105170 (263h21m31.634185302s) [noise=true , address=104.36.82.37:4684 , piece_id=64S5PKFOD6RA3NTSNECACWKM25BPY6IQ6W7LGSYESLHM63AKMECA, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.203173552 (263h21m31.634253683s) [noise=true , address=77.90.31.18:4813 , piece_id=7ABDSGFIVQ2B6V4ZORWKQJHBSWO2XBSHT3YMBCGETSP7CAGKKNFA, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.203339219 (263h21m31.63441935s) [noise=true , address=45.91.116.4:4969 , piece_id=I2XX26D4XGKO7PGLG5LA5U4WCHKWAMAUNZK2EMSM2C35YDS4VG2Q, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.203560242 (263h21m31.634640373s) [noise=true , address=104.36.82.18:4370 , piece_id=N45J6WM3GDJ7DQ4B7C3PRZD6F6EG27SCH5ZUC2AMXOSI3PZU7WNQ, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:37.203644112 (263h21m31.634724243s) [noise=true , address=45.91.116.9:4269 , piece_id=KBCSQ63A6NYTJBPEQFK4PDBK7NH7WYSIIKKCSZYNMXN33T6GD5CQ, seg_pos={0 0}, upload=273812]: Uploading piece... Aug 19 16:47:38.695836156 (263h21m33.126916277s) [upload=273812]: Next returned nil segment Aug 19 16:47:38.695852487 (263h21m33.126932618s) [upload=273812]: Waiting for error group managing segments... Aug 19 16:47:39.130935848 (263h21m33.562015979s) [noise=true , address=104.36.82.37:4684 , piece_id=64S5PKFOD6RA3NTSNECACWKM25BPY6IQ6W7LGSYESLHM63AKMECA, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.135968496 (263h21m33.567048627s) [noise=true , address=104.36.82.49:4008 , piece_id=7737UXHHDA2QNVTRMBUJWPMGCZ5CCLSQ34YIQK6ZS3L4CUMXGGPA, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.184170766 (263h21m33.615250897s) [noise=true , address=104.36.82.60:4349 , piece_id=KH2AJTUXDYN5JDJJ3KB2MZUFC5BSFZ5IMRLCC2FVO3FYVPVJXFFQ, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.196973169 (263h21m33.6280533s) [noise=true , address=104.36.82.40:4950 , piece_id=BCMHBHHHDOYRWMU2DWMK36PVWWYTPGOBW2LP3TN3V7RS7OBYUH2A, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.221372910 (263h21m33.652453041s) [noise=true , address=104.36.82.18:4370 , piece_id=N45J6WM3GDJ7DQ4B7C3PRZD6F6EG27SCH5ZUC2AMXOSI3PZU7WNQ, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.228921441 (263h21m33.660001572s) [noise=true , address=104.36.82.18:4376 , piece_id=ST4DA6OBHHBNMIU65QUDEA74KRQJYBVUKOHSMYY5Y6ZS3BAUHDUQ, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.233648886 (263h21m33.664729007s) [noise=true , address=104.36.82.6:4896 , piece_id=HSQ4SX5X5Q3F4SDXDBTDUR5XMUPZKLMXLHMO3JNCXHJB57L3BB5A, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.306907152 (263h21m33.737987283s) [noise=true , address=104.36.82.50:4024 , piece_id=ZTLIWXBXFJCQD4X4KDNCOZNWUBMN7CCQSPL36GM7YJOJFOBJ5ZBQ, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.467396807 (263h21m33.898476938s) [noise=true , address=104.36.82.51:4070 , piece_id=Y6MC5XWZBVMBBE44O2IE3FZAGUCMN4GXULPNUN2TVG4EIUQ3JMKQ, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:39.602862657 (263h21m34.033942788s) [noise=true , address=104.36.82.44:4873 , piece_id=2CAUKQ6RUCMWLY4YNDWA5LZWVFIDHIHJQH3DLNIQC4RIPFFK7Y5Q, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:40.365526145 (263h21m34.796606276s) [noise=true , address=77.90.31.17:4786 , piece_id=WXHOQG2NSD5Z53U4MH7FUAXPBTUQBP7JWH7GKZBNZZMHZHV6VLBQ, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:40.731456043 (263h21m35.162536174s) [noise=true , address=45.91.116.4:4969 , piece_id=I2XX26D4XGKO7PGLG5LA5U4WCHKWAMAUNZK2EMSM2C35YDS4VG2Q, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:41.181574439 (263h21m35.61265457s) [noise=true , address=45.91.116.8:4078 , piece_id=ONY7KHFCBUAGM7LBYVC4CEKWGYTXPYAZGKITWWQLILU7CTXICOOQ, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:41.679845880 (263h21m36.110926011s) [noise=true , address=77.90.31.18:4813 , piece_id=7ABDSGFIVQ2B6V4ZORWKQJHBSWO2XBSHT3YMBCGETSP7CAGKKNFA, seg_pos={0 0}, upload=273812]: Done uploading piece. err: <nil> Aug 19 16:47:41.679871849 (263h21m36.11095198s) [seg_pos={0 0}, upload=273812]: Segment reached optimal threshold of 14 pieces. Aug 19 16:47:41.679961461 (263h21m36.111041592s) [noise=true , address=45.91.116.11:4146 , piece_id=KGOCYLZXVMAKOBNSAVJPGADSBZWCBIVS3UBUDRYP2CYNJGXQF7QA, seg_pos={0 0}, upload=273812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp): protocol: expected piece hash; context canceled; EOF Aug 19 16:47:41.680003692 (263h21m36.111083823s) [noise=true , address=77.90.31.12:4623 , piece_id=HO6MWVF2QZ3OF6SY2UZ6ZHYUCOK6KWYFZDOV5A3YZVROU3JROEPA, seg_pos={0 0}, upload=273812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM): protocol: expected piece hash; context canceled; EOF Aug 19 16:47:41.680064949 (263h21m36.11114508s) [noise=true , address=77.90.31.11:4565 , piece_id=ALIH2EDV7RGGNR5ALESOUPXP3BGWTEO55SL4ODHQDIV2CAKQZWJA, seg_pos={0 0}, upload=273812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: expected piece hash; context canceled; EOF Aug 19 16:47:41.680122890 (263h21m36.111203021s) [noise=true , address=45.91.116.9:4269 , piece_id=KBCSQ63A6NYTJBPEQFK4PDBK7NH7WYSIIKKCSZYNMXN33T6GD5CQ, seg_pos={0 0}, upload=273812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y): protocol: expected piece hash; context canceled; EOF Aug 19 16:47:41.680155643 (263h21m36.111235774s) [seg_pos={0 0}, upload=273812]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:47:41.680231287 (263h21m36.111311418s) [upload=273812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:47:41.680235666 (263h21m36.111315797s) [upload=273812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:47:42.334122456 (263h21m36.765202587s) [upload=273812]: Done uploading segments... Aug 19 16:47:42.334140249 (263h21m36.76522038s) [upload=273812]: Upload finished. err: <nil> Duration: 8.689508692s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.87587068s FAIL (2025-08-19 16:47:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 21338s) [upload=273768]: Uploading segments... Aug 19 16:44:56.941197143 (263h18m51.372277274s) [upload=273768]: Got next segment. Inline: false Aug 19 16:44:56.941232320 (263h18m51.372312451s) [upload=273768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:44:56.941236459 (263h18m51.37231659s) [upload=273768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:44:57.179169145 (263h18m51.610249277s) [seg_pos={0 0}, upload=273768]: Begin upload segment... Aug 19 16:44:57.179239420 (263h18m51.610319551s) [seg_pos={0 0}, upload=273768]: Done begin upload segment. Aug 19 16:44:57.179269538 (263h18m51.610349669s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179307400 (263h18m51.610387551s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179784723 (263h18m51.610864854s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179905404 (263h18m51.610985535s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180182484 (263h18m51.611262616s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180645561 (263h18m51.611725682s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180924785 (263h18m51.612004916s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181130468 (263h18m51.612210599s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181461401 (263h18m51.612541522s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181689417 (263h18m51.612769548s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181988539 (263h18m51.61306867s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182215543 (263h18m51.613295674s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182547829 (263h18m51.61362796s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182768241 (263h18m51.613848372s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182984705 (263h18m51.614064836s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.183204966 (263h18m51.614285098s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:58.961741539 (263h18m53.39282167s) [upload=273768]: Next returned nil segment Aug 19 16:44:58.961758922 (263h18m53.392839054s) [upload=273768]: Waiting for error group managing segments... Aug 19 16:44:59.254505408 (263h18m53.685585529s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.280332548 (263h18m53.711412679s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.288599876 (263h18m53.719680007s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.317155488 (263h18m53.748235619s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.331253149 (263h18m53.76233328s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.346011735 (263h18m53.777091866s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.353737125 (263h18m53.784817246s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.363453003 (263h18m53.794533134s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:45:26.286817635 (263h19m20.717897766s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:40788->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286870046 (263h19m20.717950177s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:57632->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286906706 (263h19m20.717986837s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:55282->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286981619 (263h19m20.71806175s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; context deadline exceeded Aug 19 16:45:26.287080398 (263h19m20.718160529s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: rpc: dial tcp 45.91.116.8:4048: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287106397 (263h19m20.718186528s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:43594->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287139521 (263h19m20.718219652s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:45424->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287163056 (263h19m20.718243187s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:59538->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287183124 (263h19m20.718263255s) [seg_pos={0 0}, upload=273768]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:26.515324579 (263h19m20.94640471s) [upload=273768]: Done uploading segments... Aug 19 16:45:26.515351451 (263h19m20.946431582s) [upload=273768]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.875898694s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.180773322s FAIL (2025-08-19 16:47:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.180804624s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X...............X.....................] duration: 1.844438285s FAIL (2025-08-19 16:47:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 1.844461347s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.988218512s FAIL (2025-08-19 16:47:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.988239651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.815561352s FAIL (2025-08-19 16:46:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.815577525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X.X...X...X..............X......X.] duration: 5.850860822s FAIL (2025-08-19 16:46:53 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 19 16:46:05.476034332 (263h19m59.907114463s) [upload=273788]: Starting upload Aug 19 16:46:05.476083585 (263h19m59.907163716s) [upload=273788]: Done starting upload Aug 19 16:46:05.476257589 (263h19m59.90733772s) [upload=273788]: Uploading segments... Aug 19 16:46:05.476285272 (263h19m59.907365403s) [upload=273788]: Got next segment. Inline: false Aug 19 16:46:05.476308406 (263h19m59.907388527s) [upload=273788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:46:05.476312634 (263h19m59.907392765s) [upload=273788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:46:05.932526060 (263h20m0.363606191s) [seg_pos={0 0}, upload=273788]: Begin upload segment... Aug 19 16:46:05.932577498 (263h20m0.363657629s) [seg_pos={0 0}, upload=273788]: Done begin upload segment. Aug 19 16:46:05.932601444 (263h20m0.363681575s) [noise=true , address=79.127.201.215:4351 , piece_id=7QVUFFJ2WTNRJ5H4ZZZJ5IQEGZEETFYZUGHW24EQALC5IFOACEUQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.932631021 (263h20m0.363711152s) [noise=true , address=45.91.116.11:4151 , piece_id=EI7OPUWJCRPH7KV6A4554GN5XXA3WX4GKDX74JMV6H3XGEOZAJCA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.932986421 (263h20m0.364066552s) [noise=true , address=104.36.82.60:4360 , piece_id=HXCB4R5C454TGNFI4SFCIUFDFWQ6PWJW6LKUEO5UCFYLNHR2GV6A, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.933043299 (263h20m0.36412343s) [noise=true , address=104.36.82.49:4985 , piece_id=MLJXG23ELJ2O7UPKKKDDKKQM7YZBJSP6YOGJDYRF6W7PHAQB4DRQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.933253933 (263h20m0.364334064s) [noise=true , address=104.36.82.21:4478 , piece_id=WGBR5TJ65DDIFRBBAVQWE7P7GH5OCQYUID74G7VVGGO4VSYNJRSA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.933508359 (263h20m0.36458849s) [noise=true , address=77.90.31.11:4580 , piece_id=7COHLXKO53V3R5VWBCPPJXFSD52IUYQU2CD5MFMROWN5DKWVMVCA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.933877326 (263h20m0.364957457s) [noise=true , address=104.36.82.11:4145 , piece_id=YMNJ4KF3UQICNJQN42L4NIJ3W2ZG5HEZ76RHDYHAO4IXFR5CEXLQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.934175836 (263h20m0.365255967s) [noise=true , address=79.127.205.234:4594 , piece_id=JMSMCSOESK2BFVQ3ZRC3GDKGZ7Z5Q7NNCGYFO4XVT6Y2UPOAVUIA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.934408221 (263h20m0.365488352s) [noise=true , address=79.127.219.48:4095 , piece_id=BWHUWYRK4QMDY5TJA4PJLO2BRQLHSXJBM6A7K7DOXD2GHBSBRNBQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.934620778 (263h20m0.365700909s) [noise=true , address=77.90.31.21:4910 , piece_id=CI2H4O4M6N5RJISGHBFAIDEVQAVLEV46EMF4VH2F3B5X7QTIOUFA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.934815180 (263h20m0.365895311s) [noise=true , address=77.90.31.19:4856 , piece_id=YO2PXA2OA7IKPX7ENFQOABQALIPQZ65DVC2REKITGXXCYLCGDMTA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935094835 (263h20m0.366174956s) [noise=true , address=104.36.82.13:4191 , piece_id=UGDRUUTBPYAA53SVZN3OPQATJICTG57RJQ7UPEP6PFWWZEPCCH4Q, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935315056 (263h20m0.366395187s) [noise=true , address=77.90.31.12:4613 , piece_id=COJIR7HNLAUYXOCVQ6MS377HGBBRCQTP5CQWIZ4RKWNJZEIATQIQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935528515 (263h20m0.366608646s) [noise=true , address=104.36.82.50:4015 , piece_id=4T2H7SIPS3QQ6GS2634NJGONFYEVCOFVG7FRDXRVOKZGLBE7Q3JA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935767411 (263h20m0.366847542s) [noise=true , address=104.36.82.16:4313 , piece_id=R5GSQJDGRLQA7J4FSQJKGTAVAQYQJOVPOTPFQB7SWBDWU6DZEYLA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935973446 (263h20m0.367053567s) [noise=true , address=45.91.116.6:4995 , piece_id=YOVPMTTWUB37ZXKTHTLDNVG2K6PUYPY4BJL2NZVAY6TMSF6ENFHA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.936206051 (263h20m0.367286182s) [noise=true , address=77.90.31.13:4631 , piece_id=MQZP2VEBU4OJ6FMPFG5SMSGEOC7MSY4ZNNLTTBKXRB5SMFALTMSQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.936450929 (263h20m0.36753106s) [noise=true , address=104.36.82.23:4446 , piece_id=QJB3XXVYUYS6E27PGNARPOVVMUKK66NPYYXORH2KCV4XJBJG5QGQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:07.075809712 (263h20m1.506889843s) [upload=273788]: Next returned nil segment Aug 19 16:46:07.075826184 (263h20m1.506906315s) [upload=273788]: Waiting for error group managing segments... Aug 19 16:46:07.361720921 (263h20m1.792801092s) [noise=true , address=79.127.205.234:4594 , piece_id=JMSMCSOESK2BFVQ3ZRC3GDKGZ7Z5Q7NNCGYFO4XVT6Y2UPOAVUIA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.837689211 (263h20m2.268769342s) [noise=true , address=45.91.116.11:4151 , piece_id=EI7OPUWJCRPH7KV6A4554GN5XXA3WX4GKDX74JMV6H3XGEOZAJCA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.848061125 (263h20m2.279141256s) [noise=true , address=77.90.31.21:4910 , piece_id=CI2H4O4M6N5RJISGHBFAIDEVQAVLEV46EMF4VH2F3B5X7QTIOUFA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.864905961 (263h20m2.295986092s) [noise=true , address=79.127.219.48:4095 , piece_id=BWHUWYRK4QMDY5TJA4PJLO2BRQLHSXJBM6A7K7DOXD2GHBSBRNBQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.913485543 (263h20m2.344565674s) [noise=true , address=77.90.31.13:4631 , piece_id=MQZP2VEBU4OJ6FMPFG5SMSGEOC7MSY4ZNNLTTBKXRB5SMFALTMSQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.915439251 (263h20m2.346519412s) [noise=true , address=45.91.116.6:4995 , piece_id=YOVPMTTWUB37ZXKTHTLDNVG2K6PUYPY4BJL2NZVAY6TMSF6ENFHA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.916217700 (263h20m2.347297821s) [noise=true , address=77.90.31.12:4613 , piece_id=COJIR7HNLAUYXOCVQ6MS377HGBBRCQTP5CQWIZ4RKWNJZEIATQIQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.926320629 (263h20m2.35740076s) [noise=true , address=77.90.31.11:4580 , piece_id=7COHLXKO53V3R5VWBCPPJXFSD52IUYQU2CD5MFMROWN5DKWVMVCA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.978316585 (263h20m2.409396716s) [noise=true , address=79.127.201.215:4351 , piece_id=7QVUFFJ2WTNRJ5H4ZZZJ5IQEGZEETFYZUGHW24EQALC5IFOACEUQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:08.055584633 (263h20m2.486664764s) [noise=true , address=77.90.31.19:4856 , piece_id=YO2PXA2OA7IKPX7ENFQOABQALIPQZ65DVC2REKITGXXCYLCGDMTA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:10.000008046 (263h20m4.431088217s) [noise=true , address=104.36.82.23:4446 , piece_id=QJB3XXVYUYS6E27PGNARPOVVMUKK66NPYYXORH2KCV4XJBJG5QGQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:12.017472770 (263h20m6.448552901s) [noise=true , address=104.36.82.21:4478 , piece_id=WGBR5TJ65DDIFRBBAVQWE7P7GH5OCQYUID74G7VVGGO4VSYNJRSA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:13.141873356 (263h20m7.572953487s) [noise=true , address=104.36.82.16:4313 , piece_id=R5GSQJDGRLQA7J4FSQJKGTAVAQYQJOVPOTPFQB7SWBDWU6DZEYLA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:13.282446527 (263h20m7.713526648s) [noise=true , address=104.36.82.13:4191 , piece_id=UGDRUUTBPYAA53SVZN3OPQATJICTG57RJQ7UPEP6PFWWZEPCCH4Q, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:13.282470542 (263h20m7.713550673s) [seg_pos={0 0}, upload=273788]: Segment reached optimal threshold of 14 pieces. Aug 19 16:46:13.282573960 (263h20m7.713654091s) [noise=true , address=104.36.82.60:4360 , piece_id=HXCB4R5C454TGNFI4SFCIUFDFWQ6PWJW6LKUEO5UCFYLNHR2GV6A, seg_pos={0 0}, upload=273788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Aug 19 16:46:13.282617344 (263h20m7.713697475s) [noise=true , address=104.36.82.50:4015 , piece_id=4T2H7SIPS3QQ6GS2634NJGONFYEVCOFVG7FRDXRVOKZGLBE7Q3JA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Aug 19 16:46:13.282638645 (263h20m7.713718776s) [noise=true , address=104.36.82.49:4985 , piece_id=MLJXG23ELJ2O7UPKKKDDKKQM7YZBJSP6YOGJDYRF6W7PHAQB4DRQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ): protocol: expected piece hash; context canceled; EOF Aug 19 16:46:13.282689832 (263h20m7.713769953s) [noise=true , address=104.36.82.11:4145 , piece_id=YMNJ4KF3UQICNJQN42L4NIJ3W2ZG5HEZ76RHDYHAO4IXFR5CEXLQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp): protocol: read tcp 188.166.225.170:43800->104.36.82.11:4145: use of closed network connection; context canceled Aug 19 16:46:13.282726022 (263h20m7.713806143s) [seg_pos={0 0}, upload=273788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:46:13.282795384 (263h20m7.713875505s) [upload=273788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:46:13.282799492 (263h20m7.713879623s) [upload=273788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:46:13.577615256 (263h20m8.008695417s) [upload=273788]: Done uploading segments... Aug 19 16:46:13.577639713 (263h20m8.008719864s) [upload=273788]: Upload finished. err: <nil> Duration: 5.850877062s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.095973854s FAIL (2025-08-19 16:46:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.096004564s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...............X....................] duration: 1.990739573s FAIL (2025-08-19 16:46:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 1.990777126s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.710479178s FAIL (2025-08-19 16:46:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.710496802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.689740009s FAIL (2025-08-19 16:46:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 21338s) [upload=273768]: Uploading segments... Aug 19 16:44:56.941197143 (263h18m51.372277274s) [upload=273768]: Got next segment. Inline: false Aug 19 16:44:56.941232320 (263h18m51.372312451s) [upload=273768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:44:56.941236459 (263h18m51.37231659s) [upload=273768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:44:57.179169145 (263h18m51.610249277s) [seg_pos={0 0}, upload=273768]: Begin upload segment... Aug 19 16:44:57.179239420 (263h18m51.610319551s) [seg_pos={0 0}, upload=273768]: Done begin upload segment. Aug 19 16:44:57.179269538 (263h18m51.610349669s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179307400 (263h18m51.610387551s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179784723 (263h18m51.610864854s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179905404 (263h18m51.610985535s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180182484 (263h18m51.611262616s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180645561 (263h18m51.611725682s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180924785 (263h18m51.612004916s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181130468 (263h18m51.612210599s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181461401 (263h18m51.612541522s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181689417 (263h18m51.612769548s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181988539 (263h18m51.61306867s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182215543 (263h18m51.613295674s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182547829 (263h18m51.61362796s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182768241 (263h18m51.613848372s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182984705 (263h18m51.614064836s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.183204966 (263h18m51.614285098s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:58.961741539 (263h18m53.39282167s) [upload=273768]: Next returned nil segment Aug 19 16:44:58.961758922 (263h18m53.392839054s) [upload=273768]: Waiting for error group managing segments... Aug 19 16:44:59.254505408 (263h18m53.685585529s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.280332548 (263h18m53.711412679s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.288599876 (263h18m53.719680007s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.317155488 (263h18m53.748235619s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.331253149 (263h18m53.76233328s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.346011735 (263h18m53.777091866s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.353737125 (263h18m53.784817246s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.363453003 (263h18m53.794533134s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:45:26.286817635 (263h19m20.717897766s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:40788->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286870046 (263h19m20.717950177s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:57632->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286906706 (263h19m20.717986837s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:55282->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286981619 (263h19m20.71806175s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; context deadline exceeded Aug 19 16:45:26.287080398 (263h19m20.718160529s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: rpc: dial tcp 45.91.116.8:4048: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287106397 (263h19m20.718186528s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:43594->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287139521 (263h19m20.718219652s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:45424->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287163056 (263h19m20.718243187s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:59538->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287183124 (263h19m20.718263255s) [seg_pos={0 0}, upload=273768]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:26.515324579 (263h19m20.94640471s) [upload=273768]: Done uploading segments... Aug 19 16:45:26.515351451 (263h19m20.946431582s) [upload=273768]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.689776699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 26.065429856s FAIL (2025-08-19 16:46:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 21338s) [upload=273768]: Uploading segments... Aug 19 16:44:56.941197143 (263h18m51.372277274s) [upload=273768]: Got next segment. Inline: false Aug 19 16:44:56.941232320 (263h18m51.372312451s) [upload=273768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:44:56.941236459 (263h18m51.37231659s) [upload=273768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:44:57.179169145 (263h18m51.610249277s) [seg_pos={0 0}, upload=273768]: Begin upload segment... Aug 19 16:44:57.179239420 (263h18m51.610319551s) [seg_pos={0 0}, upload=273768]: Done begin upload segment. Aug 19 16:44:57.179269538 (263h18m51.610349669s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179307400 (263h18m51.610387551s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179784723 (263h18m51.610864854s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179905404 (263h18m51.610985535s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180182484 (263h18m51.611262616s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180645561 (263h18m51.611725682s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180924785 (263h18m51.612004916s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181130468 (263h18m51.612210599s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181461401 (263h18m51.612541522s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181689417 (263h18m51.612769548s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181988539 (263h18m51.61306867s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182215543 (263h18m51.613295674s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182547829 (263h18m51.61362796s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182768241 (263h18m51.613848372s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182984705 (263h18m51.614064836s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.183204966 (263h18m51.614285098s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:58.961741539 (263h18m53.39282167s) [upload=273768]: Next returned nil segment Aug 19 16:44:58.961758922 (263h18m53.392839054s) [upload=273768]: Waiting for error group managing segments... Aug 19 16:44:59.254505408 (263h18m53.685585529s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.280332548 (263h18m53.711412679s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.288599876 (263h18m53.719680007s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.317155488 (263h18m53.748235619s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.331253149 (263h18m53.76233328s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.346011735 (263h18m53.777091866s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.353737125 (263h18m53.784817246s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.363453003 (263h18m53.794533134s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:45:26.286817635 (263h19m20.717897766s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:40788->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286870046 (263h19m20.717950177s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:57632->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286906706 (263h19m20.717986837s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:55282->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286981619 (263h19m20.71806175s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; context deadline exceeded Aug 19 16:45:26.287080398 (263h19m20.718160529s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: rpc: dial tcp 45.91.116.8:4048: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287106397 (263h19m20.718186528s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:43594->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287139521 (263h19m20.718219652s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:45424->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287163056 (263h19m20.718243187s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:59538->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287183124 (263h19m20.718263255s) [seg_pos={0 0}, upload=273768]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:26.515324579 (263h19m20.94640471s) [upload=273768]: Done uploading segments... Aug 19 16:45:26.515351451 (263h19m20.946431582s) [upload=273768]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 26.065450225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 37.49516459s FAIL (2025-08-19 16:46:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 37.495185058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X.X...X...X..............X......X] duration: 9.20876488s FAIL (2025-08-19 16:46:14 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 19 16:46:05.476034332 (263h19m59.907114463s) [upload=273788]: Starting upload Aug 19 16:46:05.476083585 (263h19m59.907163716s) [upload=273788]: Done starting upload Aug 19 16:46:05.476257589 (263h19m59.90733772s) [upload=273788]: Uploading segments... Aug 19 16:46:05.476285272 (263h19m59.907365403s) [upload=273788]: Got next segment. Inline: false Aug 19 16:46:05.476308406 (263h19m59.907388527s) [upload=273788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:46:05.476312634 (263h19m59.907392765s) [upload=273788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:46:05.932526060 (263h20m0.363606191s) [seg_pos={0 0}, upload=273788]: Begin upload segment... Aug 19 16:46:05.932577498 (263h20m0.363657629s) [seg_pos={0 0}, upload=273788]: Done begin upload segment. Aug 19 16:46:05.932601444 (263h20m0.363681575s) [noise=true , address=79.127.201.215:4351 , piece_id=7QVUFFJ2WTNRJ5H4ZZZJ5IQEGZEETFYZUGHW24EQALC5IFOACEUQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.932631021 (263h20m0.363711152s) [noise=true , address=45.91.116.11:4151 , piece_id=EI7OPUWJCRPH7KV6A4554GN5XXA3WX4GKDX74JMV6H3XGEOZAJCA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.932986421 (263h20m0.364066552s) [noise=true , address=104.36.82.60:4360 , piece_id=HXCB4R5C454TGNFI4SFCIUFDFWQ6PWJW6LKUEO5UCFYLNHR2GV6A, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.933043299 (263h20m0.36412343s) [noise=true , address=104.36.82.49:4985 , piece_id=MLJXG23ELJ2O7UPKKKDDKKQM7YZBJSP6YOGJDYRF6W7PHAQB4DRQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.933253933 (263h20m0.364334064s) [noise=true , address=104.36.82.21:4478 , piece_id=WGBR5TJ65DDIFRBBAVQWE7P7GH5OCQYUID74G7VVGGO4VSYNJRSA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.933508359 (263h20m0.36458849s) [noise=true , address=77.90.31.11:4580 , piece_id=7COHLXKO53V3R5VWBCPPJXFSD52IUYQU2CD5MFMROWN5DKWVMVCA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.933877326 (263h20m0.364957457s) [noise=true , address=104.36.82.11:4145 , piece_id=YMNJ4KF3UQICNJQN42L4NIJ3W2ZG5HEZ76RHDYHAO4IXFR5CEXLQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.934175836 (263h20m0.365255967s) [noise=true , address=79.127.205.234:4594 , piece_id=JMSMCSOESK2BFVQ3ZRC3GDKGZ7Z5Q7NNCGYFO4XVT6Y2UPOAVUIA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.934408221 (263h20m0.365488352s) [noise=true , address=79.127.219.48:4095 , piece_id=BWHUWYRK4QMDY5TJA4PJLO2BRQLHSXJBM6A7K7DOXD2GHBSBRNBQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.934620778 (263h20m0.365700909s) [noise=true , address=77.90.31.21:4910 , piece_id=CI2H4O4M6N5RJISGHBFAIDEVQAVLEV46EMF4VH2F3B5X7QTIOUFA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.934815180 (263h20m0.365895311s) [noise=true , address=77.90.31.19:4856 , piece_id=YO2PXA2OA7IKPX7ENFQOABQALIPQZ65DVC2REKITGXXCYLCGDMTA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935094835 (263h20m0.366174956s) [noise=true , address=104.36.82.13:4191 , piece_id=UGDRUUTBPYAA53SVZN3OPQATJICTG57RJQ7UPEP6PFWWZEPCCH4Q, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935315056 (263h20m0.366395187s) [noise=true , address=77.90.31.12:4613 , piece_id=COJIR7HNLAUYXOCVQ6MS377HGBBRCQTP5CQWIZ4RKWNJZEIATQIQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935528515 (263h20m0.366608646s) [noise=true , address=104.36.82.50:4015 , piece_id=4T2H7SIPS3QQ6GS2634NJGONFYEVCOFVG7FRDXRVOKZGLBE7Q3JA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935767411 (263h20m0.366847542s) [noise=true , address=104.36.82.16:4313 , piece_id=R5GSQJDGRLQA7J4FSQJKGTAVAQYQJOVPOTPFQB7SWBDWU6DZEYLA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.935973446 (263h20m0.367053567s) [noise=true , address=45.91.116.6:4995 , piece_id=YOVPMTTWUB37ZXKTHTLDNVG2K6PUYPY4BJL2NZVAY6TMSF6ENFHA, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.936206051 (263h20m0.367286182s) [noise=true , address=77.90.31.13:4631 , piece_id=MQZP2VEBU4OJ6FMPFG5SMSGEOC7MSY4ZNNLTTBKXRB5SMFALTMSQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:05.936450929 (263h20m0.36753106s) [noise=true , address=104.36.82.23:4446 , piece_id=QJB3XXVYUYS6E27PGNARPOVVMUKK66NPYYXORH2KCV4XJBJG5QGQ, seg_pos={0 0}, upload=273788]: Uploading piece... Aug 19 16:46:07.075809712 (263h20m1.506889843s) [upload=273788]: Next returned nil segment Aug 19 16:46:07.075826184 (263h20m1.506906315s) [upload=273788]: Waiting for error group managing segments... Aug 19 16:46:07.361720921 (263h20m1.792801092s) [noise=true , address=79.127.205.234:4594 , piece_id=JMSMCSOESK2BFVQ3ZRC3GDKGZ7Z5Q7NNCGYFO4XVT6Y2UPOAVUIA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.837689211 (263h20m2.268769342s) [noise=true , address=45.91.116.11:4151 , piece_id=EI7OPUWJCRPH7KV6A4554GN5XXA3WX4GKDX74JMV6H3XGEOZAJCA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.848061125 (263h20m2.279141256s) [noise=true , address=77.90.31.21:4910 , piece_id=CI2H4O4M6N5RJISGHBFAIDEVQAVLEV46EMF4VH2F3B5X7QTIOUFA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.864905961 (263h20m2.295986092s) [noise=true , address=79.127.219.48:4095 , piece_id=BWHUWYRK4QMDY5TJA4PJLO2BRQLHSXJBM6A7K7DOXD2GHBSBRNBQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.913485543 (263h20m2.344565674s) [noise=true , address=77.90.31.13:4631 , piece_id=MQZP2VEBU4OJ6FMPFG5SMSGEOC7MSY4ZNNLTTBKXRB5SMFALTMSQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.915439251 (263h20m2.346519412s) [noise=true , address=45.91.116.6:4995 , piece_id=YOVPMTTWUB37ZXKTHTLDNVG2K6PUYPY4BJL2NZVAY6TMSF6ENFHA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.916217700 (263h20m2.347297821s) [noise=true , address=77.90.31.12:4613 , piece_id=COJIR7HNLAUYXOCVQ6MS377HGBBRCQTP5CQWIZ4RKWNJZEIATQIQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.926320629 (263h20m2.35740076s) [noise=true , address=77.90.31.11:4580 , piece_id=7COHLXKO53V3R5VWBCPPJXFSD52IUYQU2CD5MFMROWN5DKWVMVCA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:07.978316585 (263h20m2.409396716s) [noise=true , address=79.127.201.215:4351 , piece_id=7QVUFFJ2WTNRJ5H4ZZZJ5IQEGZEETFYZUGHW24EQALC5IFOACEUQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:08.055584633 (263h20m2.486664764s) [noise=true , address=77.90.31.19:4856 , piece_id=YO2PXA2OA7IKPX7ENFQOABQALIPQZ65DVC2REKITGXXCYLCGDMTA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:10.000008046 (263h20m4.431088217s) [noise=true , address=104.36.82.23:4446 , piece_id=QJB3XXVYUYS6E27PGNARPOVVMUKK66NPYYXORH2KCV4XJBJG5QGQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:12.017472770 (263h20m6.448552901s) [noise=true , address=104.36.82.21:4478 , piece_id=WGBR5TJ65DDIFRBBAVQWE7P7GH5OCQYUID74G7VVGGO4VSYNJRSA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:13.141873356 (263h20m7.572953487s) [noise=true , address=104.36.82.16:4313 , piece_id=R5GSQJDGRLQA7J4FSQJKGTAVAQYQJOVPOTPFQB7SWBDWU6DZEYLA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:13.282446527 (263h20m7.713526648s) [noise=true , address=104.36.82.13:4191 , piece_id=UGDRUUTBPYAA53SVZN3OPQATJICTG57RJQ7UPEP6PFWWZEPCCH4Q, seg_pos={0 0}, upload=273788]: Done uploading piece. err: <nil> Aug 19 16:46:13.282470542 (263h20m7.713550673s) [seg_pos={0 0}, upload=273788]: Segment reached optimal threshold of 14 pieces. Aug 19 16:46:13.282573960 (263h20m7.713654091s) [noise=true , address=104.36.82.60:4360 , piece_id=HXCB4R5C454TGNFI4SFCIUFDFWQ6PWJW6LKUEO5UCFYLNHR2GV6A, seg_pos={0 0}, upload=273788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Aug 19 16:46:13.282617344 (263h20m7.713697475s) [noise=true , address=104.36.82.50:4015 , piece_id=4T2H7SIPS3QQ6GS2634NJGONFYEVCOFVG7FRDXRVOKZGLBE7Q3JA, seg_pos={0 0}, upload=273788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Aug 19 16:46:13.282638645 (263h20m7.713718776s) [noise=true , address=104.36.82.49:4985 , piece_id=MLJXG23ELJ2O7UPKKKDDKKQM7YZBJSP6YOGJDYRF6W7PHAQB4DRQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ): protocol: expected piece hash; context canceled; EOF Aug 19 16:46:13.282689832 (263h20m7.713769953s) [noise=true , address=104.36.82.11:4145 , piece_id=YMNJ4KF3UQICNJQN42L4NIJ3W2ZG5HEZ76RHDYHAO4IXFR5CEXLQ, seg_pos={0 0}, upload=273788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp): protocol: read tcp 188.166.225.170:43800->104.36.82.11:4145: use of closed network connection; context canceled Aug 19 16:46:13.282726022 (263h20m7.713806143s) [seg_pos={0 0}, upload=273788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:46:13.282795384 (263h20m7.713875505s) [upload=273788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:46:13.282799492 (263h20m7.713879623s) [upload=273788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:46:13.577615256 (263h20m8.008695417s) [upload=273788]: Done uploading segments... Aug 19 16:46:13.577639713 (263h20m8.008719864s) [upload=273788]: Upload finished. err: <nil> Duration: 9.20878997s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X...............X...................] duration: 2.261138915s FAIL (2025-08-19 16:46:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.261162992s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.945715795s FAIL (2025-08-19 16:46:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.945734281s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 10.719887893s FAIL (2025-08-19 16:45:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.719899737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000751508s FAIL (2025-08-19 16:45:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000773601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X.X...X...X..............X......] duration: 51.618355304s FAIL (2025-08-19 16:45:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after fd31a5a98ef7ad7c738fc6c914c97feca16c3086fe7793213caedef2ead671c3 != before e363a3950dc7c8904e747e39e30467e9b9837d90eba82781d075dd5dfb45a310 ======LOGS====== Aug 19 16:41:13.409982568 (263h15m7.841062699s) [upload=273709]: Starting upload Aug 19 16:41:13.410028436 (263h15m7.841108567s) [upload=273709]: Done starting upload Aug 19 16:41:13.410106635 (263h15m7.841186766s) [upload=273709]: Uploading segments... Aug 19 16:41:13.410198722 (263h15m7.841278853s) [upload=273709]: Got next segment. Inline: false Aug 19 16:41:13.410229040 (263h15m7.841309171s) [upload=273709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:41:13.410251723 (263h15m7.841331855s) [upload=273709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:41:13.872381649 (263h15m8.30346178s) [seg_pos={0 0}, upload=273709]: Begin upload segment... Aug 19 16:41:13.872499073 (263h15m8.303579224s) [seg_pos={0 0}, upload=273709]: Done begin upload segment. Aug 19 16:41:13.872532007 (263h15m8.303612138s) [noise=true , address=104.36.82.51:4076 , piece_id=ZLDAWWODRACHROIHDUZBIBWW5G4P7YYTMRWLOH3CMVWL6WUXK2EA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.873251774 (263h15m8.304331905s) [noise=true , address=45.91.116.9:4274 , piece_id=X2AMQY7I7B3HCYLLXAKBDID7OHULX4VKDKGM3MP4FPK7I7CG7WMA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.873769314 (263h15m8.304849435s) [noise=true , address=45.91.116.7:4025 , piece_id=BS25QMY5LEPL66ALUNJNRD6TY2VTQWKCOV4ZF5QP3HTNN5OHVZJA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.873997080 (263h15m8.305077211s) [noise=true , address=104.36.82.6:4892 , piece_id=DCPRQIKCP4KFKHEBYZUFQF45OIMDE2UT6D5IVLKYP7CDAX7V3EPA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.874242079 (263h15m8.30532221s) [noise=true , address=77.90.31.10:4547 , piece_id=LZ72ZWLTEL2EDCMZUTTI7HUVJILT5MNRMQFTGNZNUBI6ZEYCJY5Q, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.874425910 (263h15m8.305506041s) [noise=true , address=104.36.82.59:4332 , piece_id=S6HX6BDHPXZ4VE7JCF3PCPOJIZTYTHKDNPHWNNV5MNYDGI34UBEA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.878778016 (263h15m8.309858147s) [noise=true , address=104.36.82.15:4277 , piece_id=VUVZSELHZVNC4FVE3UHRF4VMH5H3I62FU76FJZLDJL7TVX6BM4KA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.880176422 (263h15m8.311256553s) [noise=true , address=79.127.205.245:4229 , piece_id=VMPVTFRIU7RJCVYYETRVF7NZX5R47K63RZ3NBXWJBLUIT7FG75CA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.882368545 (263h15m8.313448676s) [noise=true , address=104.36.82.16:4305 , piece_id=QHX7T5HIWIHQ5ICJ26CKOWCXR5JOX3N3RTJJTAS3VGNMFEU3SUFQ, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.883649877 (263h15m8.314730008s) [noise=true , address=45.91.116.12:4245 , piece_id=RA6FALTJNYOE3QO6RE7K56MRNDL6EEVSMGBBRRVXCRRBQOKAN2JA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.884908314 (263h15m8.315988445s) [noise=true , address=104.36.82.58:4287 , piece_id=DEZRDC2MAHUV3Y4J3JJ6NIHTZJMZM6MJZFMJUJ4MTRHQROWBXABQ, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.886303414 (263h15m8.317383545s) [noise=true , address=104.36.82.23:4470 , piece_id=HHWDXFCXCDAG3SGQ5JRDP6ES6F47MAVGK57UPU7NA63TYYZSSCZQ, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.887652525 (263h15m8.318732656s) [noise=true , address=77.90.31.16:4758 , piece_id=I7LKLCPSIZHZXBT5RPYJHCYMOHE5RMU4V6S6YDUUVK26LMDF4WZA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.887980723 (263h15m8.319060854s) [noise=true , address=104.36.82.15:4271 , piece_id=CATAEC3P2IMJKOBG4R4RLOG3YF2PHLFAKD5UMEOO4UKBUNWIXFIA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.889257596 (263h15m8.320337727s) [noise=true , address=45.91.116.6:4987 , piece_id=FYRYMKH4VWX3AYXNZ25PNVCUER7CH374NBHL5PXHUQW56EDKUPNQ, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.890519520 (263h15m8.321599651s) [noise=true , address=104.36.82.34:4580 , piece_id=GVTA5YGJYHZWV5BIVKWG5Y3VYXMP4OFOVSOIU4HKFG7JIRBX7K4Q, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.891815981 (263h15m8.322896112s) [noise=true , address=104.36.82.60:4366 , piece_id=2VHHTVEWTM2MNYRTUQDWYDCRDS3KOLSIWNZ3QYHC43E7SP2RFUHA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.893091280 (263h15m8.324171411s) [noise=true , address=77.90.31.18:4835 , piece_id=POODUE3HHEYXZCRCXEPUQHTA2OGSSMM5WRXUZYW6VVKURVS27PZA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:15.025570831 (263h15m9.456650962s) [upload=273709]: Next returned nil segment Aug 19 16:41:15.025590379 (263h15m9.45667052s) [upload=273709]: Waiting for error group managing segments... Aug 19 16:41:15.333569837 (263h15m9.764649968s) [noise=true , address=79.127.205.245:4229 , piece_id=VMPVTFRIU7RJCVYYETRVF7NZX5R47K63RZ3NBXWJBLUIT7FG75CA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.789276083 (263h15m10.220356214s) [noise=true , address=45.91.116.12:4245 , piece_id=RA6FALTJNYOE3QO6RE7K56MRNDL6EEVSMGBBRRVXCRRBQOKAN2JA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.793958762 (263h15m10.225038893s) [noise=true , address=104.36.82.59:4332 , piece_id=S6HX6BDHPXZ4VE7JCF3PCPOJIZTYTHKDNPHWNNV5MNYDGI34UBEA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.808248971 (263h15m10.239329102s) [noise=true , address=45.91.116.6:4987 , piece_id=FYRYMKH4VWX3AYXNZ25PNVCUER7CH374NBHL5PXHUQW56EDKUPNQ, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.825733853 (263h15m10.256813974s) [noise=true , address=77.90.31.10:4547 , piece_id=LZ72ZWLTEL2EDCMZUTTI7HUVJILT5MNRMQFTGNZNUBI6ZEYCJY5Q, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.831514162 (263h15m10.262594293s) [noise=true , address=104.36.82.58:4287 , piece_id=DEZRDC2MAHUV3Y4J3JJ6NIHTZJMZM6MJZFMJUJ4MTRHQROWBXABQ, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.852189267 (263h15m10.283269398s) [noise=true , address=104.36.82.15:4271 , piece_id=CATAEC3P2IMJKOBG4R4RLOG3YF2PHLFAKD5UMEOO4UKBUNWIXFIA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.862765993 (263h15m10.293846124s) [noise=true , address=45.91.116.9:4274 , piece_id=X2AMQY7I7B3HCYLLXAKBDID7OHULX4VKDKGM3MP4FPK7I7CG7WMA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.885293001 (263h15m10.316373132s) [noise=true , address=104.36.82.34:4580 , piece_id=GVTA5YGJYHZWV5BIVKWG5Y3VYXMP4OFOVSOIU4HKFG7JIRBX7K4Q, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.894480999 (263h15m10.32556113s) [noise=true , address=77.90.31.16:4758 , piece_id=I7LKLCPSIZHZXBT5RPYJHCYMOHE5RMU4V6S6YDUUVK26LMDF4WZA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.924271342 (263h15m10.355351473s) [noise=true , address=104.36.82.6:4892 , piece_id=DCPRQIKCP4KFKHEBYZUFQF45OIMDE2UT6D5IVLKYP7CDAX7V3EPA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.937890448 (263h15m10.368970579s) [noise=true , address=45.91.116.7:4025 , piece_id=BS25QMY5LEPL66ALUNJNRD6TY2VTQWKCOV4ZF5QP3HTNN5OHVZJA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.960597581 (263h15m10.391677702s) [noise=true , address=104.36.82.23:4470 , piece_id=HHWDXFCXCDAG3SGQ5JRDP6ES6F47MAVGK57UPU7NA63TYYZSSCZQ, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.975606967 (263h15m10.406687098s) [noise=true , address=104.36.82.15:4277 , piece_id=VUVZSELHZVNC4FVE3UHRF4VMH5H3I62FU76FJZLDJL7TVX6BM4KA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.975633187 (263h15m10.406713318s) [seg_pos={0 0}, upload=273709]: Segment reached optimal threshold of 14 pieces. Aug 19 16:41:15.975749901 (263h15m10.406830042s) [noise=true , address=104.36.82.16:4305 , piece_id=QHX7T5HIWIHQ5ICJ26CKOWCXR5JOX3N3RTJJTAS3VGNMFEU3SUFQ, seg_pos={0 0}, upload=273709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZMhNJYCNg8K16ZqsVTLBCwGn51vtG5u56cxcqgZPfQaJxgNZv): protocol: expected piece hash; context canceled; EOF Aug 19 16:41:15.975865903 (263h15m10.406946034s) [noise=true , address=77.90.31.18:4835 , piece_id=POODUE3HHEYXZCRCXEPUQHTA2OGSSMM5WRXUZYW6VVKURVS27PZA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg): protocol: expected piece hash; context canceled; EOF Aug 19 16:41:15.975918513 (263h15m10.406998644s) [noise=true , address=104.36.82.51:4076 , piece_id=ZLDAWWODRACHROIHDUZBIBWW5G4P7YYTMRWLOH3CMVWL6WUXK2EA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Aug 19 16:41:15.975959201 (263h15m10.407039332s) [noise=true , address=104.36.82.60:4366 , piece_id=2VHHTVEWTM2MNYRTUQDWYDCRDS3KOLSIWNZ3QYHC43E7SP2RFUHA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:54284->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:54284->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54284->104.36.82.60:4366: use of closed network connection Aug 19 16:41:15.975972066 (263h15m10.407052197s) [seg_pos={0 0}, upload=273709]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:41:15.976032321 (263h15m10.407112452s) [upload=273709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:41:15.976036689 (263h15m10.40711682s) [upload=273709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:41:16.461966026 (263h15m10.893046157s) [upload=273709]: Done uploading segments... Aug 19 16:41:16.461986625 (263h15m10.893066756s) [upload=273709]: Upload finished. err: <nil> Duration: 51.618373268s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.229652726s FAIL (2025-08-19 16:45:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 263h18m33.75518017s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324344006 (263h18m33.755424127s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324633220 (263h18m33.755713351s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.324851286 (263h18m33.755931417s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325068942 (263h18m33.756149063s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325283613 (263h18m33.756363744s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325499857 (263h18m33.756579998s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.325759794 (263h18m33.756839915s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326059428 (263h18m33.757139549s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326277565 (263h18m33.757357696s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326470835 (263h18m33.757550966s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326673753 (263h18m33.757753884s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:39.326867664 (263h18m33.757947795s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Uploading piece... Aug 19 16:44:40.851885389 (263h18m35.28296552s) [upload=273764]: Next returned nil segment Aug 19 16:44:40.851903334 (263h18m35.282983465s) [upload=273764]: Waiting for error group managing segments... Aug 19 16:44:41.244668372 (263h18m35.675748503s) [noise=true , address=104.36.82.36:4650 , piece_id=6JMI5PHK3YXL7ETF3LGWLWEQYKPYKGTESBOGCUSYUZ4E7G6LHGWQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.277280365 (263h18m35.708360496s) [noise=true , address=104.36.82.56:4243 , piece_id=CKIAF43JIMZXTEFSPQ64DO7WXTKF3367TRU45FKBLK5Z2OHOGLUA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.328561665 (263h18m35.759641786s) [noise=true , address=104.36.82.60:4354 , piece_id=QLYNM643AGZAUXQBZC6F5H5BSLL2E24AWQSRWM3AOEZCRNIQDNSQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.331747139 (263h18m35.76282727s) [noise=true , address=79.127.201.214:4322 , piece_id=TTN6TTENBCQG2H7TMAXEE7Q3H4GNPC2E75PMD3OB2Y3HBTP672FA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.357720320 (263h18m35.788800481s) [noise=true , address=104.36.82.34:4576 , piece_id=H3GRBD2FX5DMY5UV62LTYC2CAT2ONUSWUM7C2CTC6HQAD3BVM6DA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.378190805 (263h18m35.809270936s) [noise=true , address=104.36.82.23:4480 , piece_id=KDE2R7YVWYV2GQE4J5MBQRWL32OPNOO5MJ43OK2JLIK4OVKRPBCA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:44:41.486480333 (263h18m35.917560464s) [noise=true , address=104.36.82.18:4391 , piece_id=4YNMDY2IFOLME2PYZ6EUFK2VSIPM42IP5SX75NFZQ2TSOY3XMIGQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: <nil> Aug 19 16:45:38.230570787 (263h19m32.661650918s) [noise=true , address=77.90.31.12:4595 , piece_id=BIC3B3LKRH4VQ7TFUXMGFHR25ANBQ3NGF3KEQS3IEOXAMNL7PT7A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: expected piece hash; context deadline exceeded; EOF Aug 19 16:45:38.230979869 (263h19m32.66206s) [noise=true , address=77.90.31.18:4840 , piece_id=JWOETSFSZ3HELLZNLEASYNGW5OR64TMQVOFYYJAJVTHWCBHVTSLQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 188.166.225.170:39946->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231017812 (263h19m32.662097943s) [noise=true , address=45.91.116.6:4983 , piece_id=F3TJ5RWTFWETPMGZJTXABGYQMUHD3GGOR25YV5WMVLTTV7FTYTJQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, address:45.91.116.6:4983): protocol: read tcp 188.166.225.170:44164->45.91.116.6:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231063489 (263h19m32.66214362s) [noise=true , address=45.91.116.14:4203 , piece_id=WRLBES3FI5EA5AQLZXJYGDROP3KBG3E2F22R4YD6A5KKI44PLV2Q, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12UBvE64jDPhNfGqS1Zj9cB5DJH8NjBfCgPKbWRRzMRHFd1xxju, address:45.91.116.14:4203): protocol: write tcp 188.166.225.170:51738->45.91.116.14:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231090952 (263h19m32.662171083s) [noise=true , address=77.90.31.15:4707 , piece_id=G2RZQMQKQ6Y44DLP5CC5Z32G5CHIIQXG33YHJRXVMJ5EQDZTZKBA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: read tcp 188.166.225.170:57158->77.90.31.15:4707: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231110719 (263h19m32.66219085s) [noise=true , address=77.90.31.21:4902 , piece_id=JBWV3IMLG36WEBTZOBH2LKZSRAANKCLJUYI5NYLYSUJ3QQX6NWCQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:50784->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231140165 (263h19m32.662220296s) [noise=true , address=77.90.31.16:4749 , piece_id=XCPXVBYLQH345FPIW4KDNLBFUCDDEW2H4B25WIXGOO5FVMAIVKPQ, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: read tcp 188.166.225.170:51668->77.90.31.16:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231144814 (263h19m32.662224945s) [noise=true , address=77.90.31.19:4865 , piece_id=B3HUFIJZKVZJH4OBBO2DD7S3I7HX4LO2IQ74XJPO4DTW4QER35NA, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: read tcp 188.166.225.170:39398->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231177167 (263h19m32.662257298s) [noise=true , address=45.91.116.5:4324 , piece_id=CKOXA5M5GXKD4L4TBJUWGK7RZITLVGPYAYEIP673PMYSUNUJZR6A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: read tcp 188.166.225.170:58526->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231184571 (263h19m32.662264702s) [noise=true , address=45.91.116.12:4236 , piece_id=LCRQS2GOSHOSLZBGQTGB3Q3PA4VN7KXNW66TDZ5O424TPJWTQE3A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:1WpwT11kNqEF8CHQ4uhajANLLyDi5a9gCh9eQMKdE1tYjkbhyH, address:45.91.116.12:4236): protocol: read tcp 188.166.225.170:57166->45.91.116.12:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231189770 (263h19m32.662269901s) [noise=true , address=45.91.116.11:4142 , piece_id=OOHUJ55RVDXCXHIJ6P44LBJPHAG7JDUSABRXHKYPXRI4OPEJSV4A, seg_pos={0 0}, upload=273764]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: read tcp 188.166.225.170:54242->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:38.231215850 (263h19m32.662295981s) [seg_pos={0 0}, upload=273764]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 19 16:45:38.459100173 (263h19m32.890180304s) [upload=273764]: Done uploading segments... Aug 19 16:45:38.459128316 (263h19m32.890208447s) [upload=273764]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.229678716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.230581108s FAIL (2025-08-19 16:45:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 21338s) [upload=273768]: Uploading segments... Aug 19 16:44:56.941197143 (263h18m51.372277274s) [upload=273768]: Got next segment. Inline: false Aug 19 16:44:56.941232320 (263h18m51.372312451s) [upload=273768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:44:56.941236459 (263h18m51.37231659s) [upload=273768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:44:57.179169145 (263h18m51.610249277s) [seg_pos={0 0}, upload=273768]: Begin upload segment... Aug 19 16:44:57.179239420 (263h18m51.610319551s) [seg_pos={0 0}, upload=273768]: Done begin upload segment. Aug 19 16:44:57.179269538 (263h18m51.610349669s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179307400 (263h18m51.610387551s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179784723 (263h18m51.610864854s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.179905404 (263h18m51.610985535s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180182484 (263h18m51.611262616s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180645561 (263h18m51.611725682s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.180924785 (263h18m51.612004916s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181130468 (263h18m51.612210599s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181461401 (263h18m51.612541522s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181689417 (263h18m51.612769548s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.181988539 (263h18m51.61306867s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182215543 (263h18m51.613295674s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182547829 (263h18m51.61362796s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182768241 (263h18m51.613848372s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.182984705 (263h18m51.614064836s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:57.183204966 (263h18m51.614285098s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Uploading piece... Aug 19 16:44:58.961741539 (263h18m53.39282167s) [upload=273768]: Next returned nil segment Aug 19 16:44:58.961758922 (263h18m53.392839054s) [upload=273768]: Waiting for error group managing segments... Aug 19 16:44:59.254505408 (263h18m53.685585529s) [noise=true , address=104.36.82.7:4910 , piece_id=EFIOQFRJEA3TOTCRQBUPVOOMUK5AG2DXLS7DVQ2CAMTBA2QRES3Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.280332548 (263h18m53.711412679s) [noise=true , address=104.36.82.14:4223 , piece_id=4OOBCI62RYOCJURJ52PEHHKHRY7WSKADX6YTQKLFZZMAYKOYXAOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.288599876 (263h18m53.719680007s) [noise=true , address=104.36.82.8:4969 , piece_id=6OFZ7CB5S4BPKP6QEBBMCTOA2ZEAMSFX4U7KRIIBXWHG7GDQGJOQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.317155488 (263h18m53.748235619s) [noise=true , address=104.36.82.13:4187 , piece_id=ONEJ33MWGDMN3QRGCPHDAW3ZFIPZH6RQGFSQKRXBTD6X7QVH3VYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.331253149 (263h18m53.76233328s) [noise=true , address=104.36.82.5:4839 , piece_id=WKRI2VJMCT5VY2RBPNWARVYDXWZPCJHLNI4HOX2GM7OZEZOMIO2Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.346011735 (263h18m53.777091866s) [noise=true , address=104.36.82.8:4970 , piece_id=XSWJGTBNSPI47F7A6VH7AWP6ATDH4AKMIWF25ZI2DALPNSLJBVXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.353737125 (263h18m53.784817246s) [noise=true , address=104.36.82.4:4804 , piece_id=MHN7MNW26GFZ2KGYBSYWDCV2AWMJ7IQPSEMVM7CGJOIHMIWQJPXQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:44:59.363453003 (263h18m53.794533134s) [noise=true , address=104.36.82.9:4005 , piece_id=ECSBHGXV76RV5GQMU4OQFUSOJZHL7APRDE3TMA46PUGDR62EWFSA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: <nil> Aug 19 16:45:26.286817635 (263h19m20.717897766s) [noise=true , address=45.91.116.5:4299 , piece_id=ZUT5UDVJGO2Q7LGR7AKADJGIZBWGEW7UAW7TU4F7X4SQTGQL6AZA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:40788->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286870046 (263h19m20.717950177s) [noise=true , address=45.91.116.12:4226 , piece_id=PAQWDG36D5EPZBREKDBCSM5OMGVCSZMMK5SQIJ4XACYBZIR32GYA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: read tcp 188.166.225.170:57632->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286906706 (263h19m20.717986837s) [noise=true , address=77.90.31.15:4699 , piece_id=VS4MPCACD3C6J75F556HAYROSBISZXBWFJCMH3AKWPULJ7RVXZZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:55282->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.286981619 (263h19m20.71806175s) [noise=true , address=77.90.31.10:4519 , piece_id=UPPGJWBJHSGAKG4V4X7QJTZJEKIGD4IH2PVNWOYWQDCR6F3LTDZQ, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; context deadline exceeded Aug 19 16:45:26.287080398 (263h19m20.718160529s) [noise=true , address=45.91.116.8:4048 , piece_id=ENF77NFR725TZUFLQXJOMHGRHARNNHWZBM5SSPBE4PLVRNS2ACJA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: rpc: dial tcp 45.91.116.8:4048: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287106397 (263h19m20.718186528s) [noise=true , address=77.90.31.18:4807 , piece_id=LLFZG7YI24EXQUZLFS4T6OD2HRUKNGV3ANFGSGPKBWP575UT4C7Q, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:43594->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287139521 (263h19m20.718219652s) [noise=true , address=45.91.116.4:4939 , piece_id=NFMGN7HW3LXKFSS7UDTO5YDGYK3H2BE7Y3CYCDS5GWQ3A5YWSFWA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 188.166.225.170:45424->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287163056 (263h19m20.718243187s) [noise=true , address=45.91.116.6:4974 , piece_id=SBXEN64JN455665XQUFCFRVJL37LSHUVLEZNDXTGJ7JM3LPJO5RA, seg_pos={0 0}, upload=273768]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:59538->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:45:26.287183124 (263h19m20.718263255s) [seg_pos={0 0}, upload=273768]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 19 16:45:26.515324579 (263h19m20.94640471s) [upload=273768]: Done uploading segments... Aug 19 16:45:26.515351451 (263h19m20.946431582s) [upload=273768]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.230607537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X........................X...............X..................] duration: 2.007310138s FAIL (2025-08-19 16:45:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 2.007344613s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001258154s FAIL (2025-08-19 16:45:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001279545s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X.X...X...X..............X.....] duration: 5.844150625s FAIL (2025-08-19 16:44:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after fd31a5a98ef7ad7c738fc6c914c97feca16c3086fe7793213caedef2ead671c3 != before e363a3950dc7c8904e747e39e30467e9b9837d90eba82781d075dd5dfb45a310 ======LOGS====== Aug 19 16:41:13.409982568 (263h15m7.841062699s) [upload=273709]: Starting upload Aug 19 16:41:13.410028436 (263h15m7.841108567s) [upload=273709]: Done starting upload Aug 19 16:41:13.410106635 (263h15m7.841186766s) [upload=273709]: Uploading segments... Aug 19 16:41:13.410198722 (263h15m7.841278853s) [upload=273709]: Got next segment. Inline: false Aug 19 16:41:13.410229040 (263h15m7.841309171s) [upload=273709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:41:13.410251723 (263h15m7.841331855s) [upload=273709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:41:13.872381649 (263h15m8.30346178s) [seg_pos={0 0}, upload=273709]: Begin upload segment... Aug 19 16:41:13.872499073 (263h15m8.303579224s) [seg_pos={0 0}, upload=273709]: Done begin upload segment. Aug 19 16:41:13.872532007 (263h15m8.303612138s) [noise=true , address=104.36.82.51:4076 , piece_id=ZLDAWWODRACHROIHDUZBIBWW5G4P7YYTMRWLOH3CMVWL6WUXK2EA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.873251774 (263h15m8.304331905s) [noise=true , address=45.91.116.9:4274 , piece_id=X2AMQY7I7B3HCYLLXAKBDID7OHULX4VKDKGM3MP4FPK7I7CG7WMA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.873769314 (263h15m8.304849435s) [noise=true , address=45.91.116.7:4025 , piece_id=BS25QMY5LEPL66ALUNJNRD6TY2VTQWKCOV4ZF5QP3HTNN5OHVZJA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.873997080 (263h15m8.305077211s) [noise=true , address=104.36.82.6:4892 , piece_id=DCPRQIKCP4KFKHEBYZUFQF45OIMDE2UT6D5IVLKYP7CDAX7V3EPA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.874242079 (263h15m8.30532221s) [noise=true , address=77.90.31.10:4547 , piece_id=LZ72ZWLTEL2EDCMZUTTI7HUVJILT5MNRMQFTGNZNUBI6ZEYCJY5Q, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.874425910 (263h15m8.305506041s) [noise=true , address=104.36.82.59:4332 , piece_id=S6HX6BDHPXZ4VE7JCF3PCPOJIZTYTHKDNPHWNNV5MNYDGI34UBEA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.878778016 (263h15m8.309858147s) [noise=true , address=104.36.82.15:4277 , piece_id=VUVZSELHZVNC4FVE3UHRF4VMH5H3I62FU76FJZLDJL7TVX6BM4KA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.880176422 (263h15m8.311256553s) [noise=true , address=79.127.205.245:4229 , piece_id=VMPVTFRIU7RJCVYYETRVF7NZX5R47K63RZ3NBXWJBLUIT7FG75CA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.882368545 (263h15m8.313448676s) [noise=true , address=104.36.82.16:4305 , piece_id=QHX7T5HIWIHQ5ICJ26CKOWCXR5JOX3N3RTJJTAS3VGNMFEU3SUFQ, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.883649877 (263h15m8.314730008s) [noise=true , address=45.91.116.12:4245 , piece_id=RA6FALTJNYOE3QO6RE7K56MRNDL6EEVSMGBBRRVXCRRBQOKAN2JA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.884908314 (263h15m8.315988445s) [noise=true , address=104.36.82.58:4287 , piece_id=DEZRDC2MAHUV3Y4J3JJ6NIHTZJMZM6MJZFMJUJ4MTRHQROWBXABQ, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.886303414 (263h15m8.317383545s) [noise=true , address=104.36.82.23:4470 , piece_id=HHWDXFCXCDAG3SGQ5JRDP6ES6F47MAVGK57UPU7NA63TYYZSSCZQ, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.887652525 (263h15m8.318732656s) [noise=true , address=77.90.31.16:4758 , piece_id=I7LKLCPSIZHZXBT5RPYJHCYMOHE5RMU4V6S6YDUUVK26LMDF4WZA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.887980723 (263h15m8.319060854s) [noise=true , address=104.36.82.15:4271 , piece_id=CATAEC3P2IMJKOBG4R4RLOG3YF2PHLFAKD5UMEOO4UKBUNWIXFIA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.889257596 (263h15m8.320337727s) [noise=true , address=45.91.116.6:4987 , piece_id=FYRYMKH4VWX3AYXNZ25PNVCUER7CH374NBHL5PXHUQW56EDKUPNQ, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.890519520 (263h15m8.321599651s) [noise=true , address=104.36.82.34:4580 , piece_id=GVTA5YGJYHZWV5BIVKWG5Y3VYXMP4OFOVSOIU4HKFG7JIRBX7K4Q, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.891815981 (263h15m8.322896112s) [noise=true , address=104.36.82.60:4366 , piece_id=2VHHTVEWTM2MNYRTUQDWYDCRDS3KOLSIWNZ3QYHC43E7SP2RFUHA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:13.893091280 (263h15m8.324171411s) [noise=true , address=77.90.31.18:4835 , piece_id=POODUE3HHEYXZCRCXEPUQHTA2OGSSMM5WRXUZYW6VVKURVS27PZA, seg_pos={0 0}, upload=273709]: Uploading piece... Aug 19 16:41:15.025570831 (263h15m9.456650962s) [upload=273709]: Next returned nil segment Aug 19 16:41:15.025590379 (263h15m9.45667052s) [upload=273709]: Waiting for error group managing segments... Aug 19 16:41:15.333569837 (263h15m9.764649968s) [noise=true , address=79.127.205.245:4229 , piece_id=VMPVTFRIU7RJCVYYETRVF7NZX5R47K63RZ3NBXWJBLUIT7FG75CA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.789276083 (263h15m10.220356214s) [noise=true , address=45.91.116.12:4245 , piece_id=RA6FALTJNYOE3QO6RE7K56MRNDL6EEVSMGBBRRVXCRRBQOKAN2JA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.793958762 (263h15m10.225038893s) [noise=true , address=104.36.82.59:4332 , piece_id=S6HX6BDHPXZ4VE7JCF3PCPOJIZTYTHKDNPHWNNV5MNYDGI34UBEA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.808248971 (263h15m10.239329102s) [noise=true , address=45.91.116.6:4987 , piece_id=FYRYMKH4VWX3AYXNZ25PNVCUER7CH374NBHL5PXHUQW56EDKUPNQ, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.825733853 (263h15m10.256813974s) [noise=true , address=77.90.31.10:4547 , piece_id=LZ72ZWLTEL2EDCMZUTTI7HUVJILT5MNRMQFTGNZNUBI6ZEYCJY5Q, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.831514162 (263h15m10.262594293s) [noise=true , address=104.36.82.58:4287 , piece_id=DEZRDC2MAHUV3Y4J3JJ6NIHTZJMZM6MJZFMJUJ4MTRHQROWBXABQ, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.852189267 (263h15m10.283269398s) [noise=true , address=104.36.82.15:4271 , piece_id=CATAEC3P2IMJKOBG4R4RLOG3YF2PHLFAKD5UMEOO4UKBUNWIXFIA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.862765993 (263h15m10.293846124s) [noise=true , address=45.91.116.9:4274 , piece_id=X2AMQY7I7B3HCYLLXAKBDID7OHULX4VKDKGM3MP4FPK7I7CG7WMA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.885293001 (263h15m10.316373132s) [noise=true , address=104.36.82.34:4580 , piece_id=GVTA5YGJYHZWV5BIVKWG5Y3VYXMP4OFOVSOIU4HKFG7JIRBX7K4Q, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.894480999 (263h15m10.32556113s) [noise=true , address=77.90.31.16:4758 , piece_id=I7LKLCPSIZHZXBT5RPYJHCYMOHE5RMU4V6S6YDUUVK26LMDF4WZA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.924271342 (263h15m10.355351473s) [noise=true , address=104.36.82.6:4892 , piece_id=DCPRQIKCP4KFKHEBYZUFQF45OIMDE2UT6D5IVLKYP7CDAX7V3EPA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.937890448 (263h15m10.368970579s) [noise=true , address=45.91.116.7:4025 , piece_id=BS25QMY5LEPL66ALUNJNRD6TY2VTQWKCOV4ZF5QP3HTNN5OHVZJA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.960597581 (263h15m10.391677702s) [noise=true , address=104.36.82.23:4470 , piece_id=HHWDXFCXCDAG3SGQ5JRDP6ES6F47MAVGK57UPU7NA63TYYZSSCZQ, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.975606967 (263h15m10.406687098s) [noise=true , address=104.36.82.15:4277 , piece_id=VUVZSELHZVNC4FVE3UHRF4VMH5H3I62FU76FJZLDJL7TVX6BM4KA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: <nil> Aug 19 16:41:15.975633187 (263h15m10.406713318s) [seg_pos={0 0}, upload=273709]: Segment reached optimal threshold of 14 pieces. Aug 19 16:41:15.975749901 (263h15m10.406830042s) [noise=true , address=104.36.82.16:4305 , piece_id=QHX7T5HIWIHQ5ICJ26CKOWCXR5JOX3N3RTJJTAS3VGNMFEU3SUFQ, seg_pos={0 0}, upload=273709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZMhNJYCNg8K16ZqsVTLBCwGn51vtG5u56cxcqgZPfQaJxgNZv): protocol: expected piece hash; context canceled; EOF Aug 19 16:41:15.975865903 (263h15m10.406946034s) [noise=true , address=77.90.31.18:4835 , piece_id=POODUE3HHEYXZCRCXEPUQHTA2OGSSMM5WRXUZYW6VVKURVS27PZA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg): protocol: expected piece hash; context canceled; EOF Aug 19 16:41:15.975918513 (263h15m10.406998644s) [noise=true , address=104.36.82.51:4076 , piece_id=ZLDAWWODRACHROIHDUZBIBWW5G4P7YYTMRWLOH3CMVWL6WUXK2EA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Aug 19 16:41:15.975959201 (263h15m10.407039332s) [noise=true , address=104.36.82.60:4366 , piece_id=2VHHTVEWTM2MNYRTUQDWYDCRDS3KOLSIWNZ3QYHC43E7SP2RFUHA, seg_pos={0 0}, upload=273709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:54284->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:54284->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54284->104.36.82.60:4366: use of closed network connection Aug 19 16:41:15.975972066 (263h15m10.407052197s) [seg_pos={0 0}, upload=273709]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:41:15.976032321 (263h15m10.407112452s) [upload=273709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:41:15.976036689 (263h15m10.40711682s) [upload=273709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:41:16.461966026 (263h15m10.893046157s) [upload=273709]: Done uploading segments... Aug 19 16:41:16.461986625 (263h15m10.893066756s) [upload=273709]: Upload finished. err: <nil> Duration: 5.844184811s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X........................X...............X.................] duration: 1.900199805s FAIL (2025-08-19 16:44:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 1.900221185s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X........................X...............X................] duration: 1.863953229s FAIL (2025-08-19 16:43:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after be5a5f4d20475bc6e6586ca4cb41bd260785c25704d0f801265da5a966efbd34 != before fab796bcbc8cefce4dfe0fedb7e23ecb2129705e75b1b66c720b4c58f0a82c58 ======LOGS====== Aug 19 16:31:34.462024854 (263h5m28.893104995s) [upload=273541]: Starting upload Aug 19 16:31:34.462073225 (263h5m28.893153356s) [upload=273541]: Done starting upload Aug 19 16:31:34.462124303 (263h5m28.893204434s) [upload=273541]: Uploading segments... Aug 19 16:31:34.462159490 (263h5m28.893239621s) [upload=273541]: Got next segment. Inline: true Aug 19 16:31:34.462163108 (263h5m28.893243239s) [upload=273541]: Waiting for error group managing segments... Aug 19 16:31:34.462196912 (263h5m28.893277044s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:31:34.462200970 (263h5m28.893281101s) [upload=273541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 16:31:34.462203415 (263h5m28.893283546s) [upload=273541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:31:34.760143563 (263h5m29.191223694s) [upload=273541]: Done uploading segments... Aug 19 16:31:34.760161496 (263h5m29.191241628s) [upload=273541]: Upload finished. err: <nil> Duration:* 1.863984399s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)