Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.409395291s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:40:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.409483359s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.......] duration: 6.047191822s trace: https://jaeger.dev.storj.io/trace/7aeea27d43bad8d4 FAIL (2025-12-28 08:40:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q) context deadline exceeded ======LOGS====== Dec 28 08:34:53.406660854 (545h5m53.277743889s) [upload=478008]: Starting upload Dec 28 08:34:53.406730828 (545h5m53.277813873s) [upload=478008]: Done starting upload Dec 28 08:34:53.406852530 (545h5m53.277935565s) [upload=478008]: Uploading segments... Dec 28 08:34:53.406999069 (545h5m53.278082114s) [upload=478008]: Got next segment. Inline: false Dec 28 08:34:53.407031791 (545h5m53.278114836s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:53.407060696 (545h5m53.278143741s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:53.655052049 (545h5m53.526135114s) [seg_pos={0 0}, upload=478008]: Begin upload segment... Dec 28 08:34:53.655107123 (545h5m53.526190168s) [seg_pos={0 0}, upload=478008]: Done begin upload segment. Dec 28 08:34:53.655162619 (545h5m53.526245664s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655175684 (545h5m53.526258729s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655926524 (545h5m53.527009569s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656153545 (545h5m53.527236591s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656339580 (545h5m53.527422615s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656657405 (545h5m53.52774045s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657123973 (545h5m53.528207019s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657446227 (545h5m53.528529262s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658059966 (545h5m53.529143001s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658479505 (545h5m53.52956254s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659150342 (545h5m53.530233387s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659482926 (545h5m53.530565971s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659605789 (545h5m53.530688834s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659895872 (545h5m53.530978907s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660306494 (545h5m53.531389529s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660662522 (545h5m53.531745567s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:55.372249885 (545h5m55.24333292s) [upload=478008]: Next returned nil segment Dec 28 08:34:55.372267248 (545h5m55.243350293s) [upload=478008]: Waiting for error group managing segments... Dec 28 08:34:55.659647340 (545h5m55.530730385s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.661694348 (545h5m55.532777383s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.663307369 (545h5m55.534390414s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.664639025 (545h5m55.53572206s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.671168919 (545h5m55.542251964s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.672574496 (545h5m55.543657531s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.673971776 (545h5m55.545054811s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680441596 (545h5m55.551524631s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680521839 (545h5m55.551604874s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.712067176 (545h5m55.583150221s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.940884955 (545h5m55.811967991s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.948922120 (545h5m55.820005155s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068098813 (545h5m55.939181848s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068123571 (545h5m55.939206616s) [seg_pos={0 0}, upload=478008]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:56.068349130 (545h5m55.939432175s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068358368 (545h5m55.939441403s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068646216 (545h5m55.939729261s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection Dec 28 08:34:56.068654592 (545h5m55.939737637s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068667766 (545h5m55.939750811s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068676033 (545h5m55.939759078s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068792815 (545h5m55.93987586s) [seg_pos={0 0}, upload=478008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:56.068880953 (545h5m55.939963988s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:56.068886644 (545h5m55.939969689s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:56.350764128 (545h5m56.221847183s) [upload=478008]: Done uploading segments... Dec 28 08:34:56.350785848 (545h5m56.221868883s) [upload=478008]: Upload finished. err: <nil> Duration: 6.047223143s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 8.248264822s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:39:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.24829113s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX....X] duration: 5.001326532s trace: https://jaeger.dev.storj.io/trace/37411f0c9caac311 FAIL (2025-12-28 08:39:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001349446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX......] duration: 5.799317106s trace: https://jaeger.dev.storj.io/trace/7aeea27d43bad8d4 FAIL (2025-12-28 08:39:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q) context deadline exceeded ======LOGS====== Dec 28 08:34:53.406660854 (545h5m53.277743889s) [upload=478008]: Starting upload Dec 28 08:34:53.406730828 (545h5m53.277813873s) [upload=478008]: Done starting upload Dec 28 08:34:53.406852530 (545h5m53.277935565s) [upload=478008]: Uploading segments... Dec 28 08:34:53.406999069 (545h5m53.278082114s) [upload=478008]: Got next segment. Inline: false Dec 28 08:34:53.407031791 (545h5m53.278114836s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:53.407060696 (545h5m53.278143741s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:53.655052049 (545h5m53.526135114s) [seg_pos={0 0}, upload=478008]: Begin upload segment... Dec 28 08:34:53.655107123 (545h5m53.526190168s) [seg_pos={0 0}, upload=478008]: Done begin upload segment. Dec 28 08:34:53.655162619 (545h5m53.526245664s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655175684 (545h5m53.526258729s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655926524 (545h5m53.527009569s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656153545 (545h5m53.527236591s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656339580 (545h5m53.527422615s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656657405 (545h5m53.52774045s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657123973 (545h5m53.528207019s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657446227 (545h5m53.528529262s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658059966 (545h5m53.529143001s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658479505 (545h5m53.52956254s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659150342 (545h5m53.530233387s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659482926 (545h5m53.530565971s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659605789 (545h5m53.530688834s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659895872 (545h5m53.530978907s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660306494 (545h5m53.531389529s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660662522 (545h5m53.531745567s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:55.372249885 (545h5m55.24333292s) [upload=478008]: Next returned nil segment Dec 28 08:34:55.372267248 (545h5m55.243350293s) [upload=478008]: Waiting for error group managing segments... Dec 28 08:34:55.659647340 (545h5m55.530730385s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.661694348 (545h5m55.532777383s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.663307369 (545h5m55.534390414s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.664639025 (545h5m55.53572206s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.671168919 (545h5m55.542251964s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.672574496 (545h5m55.543657531s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.673971776 (545h5m55.545054811s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680441596 (545h5m55.551524631s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680521839 (545h5m55.551604874s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.712067176 (545h5m55.583150221s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.940884955 (545h5m55.811967991s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.948922120 (545h5m55.820005155s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068098813 (545h5m55.939181848s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068123571 (545h5m55.939206616s) [seg_pos={0 0}, upload=478008]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:56.068349130 (545h5m55.939432175s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068358368 (545h5m55.939441403s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068646216 (545h5m55.939729261s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection Dec 28 08:34:56.068654592 (545h5m55.939737637s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068667766 (545h5m55.939750811s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068676033 (545h5m55.939759078s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068792815 (545h5m55.93987586s) [seg_pos={0 0}, upload=478008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:56.068880953 (545h5m55.939963988s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:56.068886644 (545h5m55.939969689s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:56.350764128 (545h5m56.221847183s) [upload=478008]: Done uploading segments... Dec 28 08:34:56.350785848 (545h5m56.221868883s) [upload=478008]: Upload finished. err: <nil> Duration: 5.799348506s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX....] duration: 3.583858644s trace: https://jaeger.dev.storj.io/trace/58488bbea5e068ff FAIL (2025-12-28 08:39:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.583893189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 15.67023407s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:38:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 15.670259399s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.....] duration: 6.152972147s trace: https://jaeger.dev.storj.io/trace/7aeea27d43bad8d4 FAIL (2025-12-28 08:38:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q) context deadline exceeded ======LOGS====== Dec 28 08:34:53.406660854 (545h5m53.277743889s) [upload=478008]: Starting upload Dec 28 08:34:53.406730828 (545h5m53.277813873s) [upload=478008]: Done starting upload Dec 28 08:34:53.406852530 (545h5m53.277935565s) [upload=478008]: Uploading segments... Dec 28 08:34:53.406999069 (545h5m53.278082114s) [upload=478008]: Got next segment. Inline: false Dec 28 08:34:53.407031791 (545h5m53.278114836s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:53.407060696 (545h5m53.278143741s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:53.655052049 (545h5m53.526135114s) [seg_pos={0 0}, upload=478008]: Begin upload segment... Dec 28 08:34:53.655107123 (545h5m53.526190168s) [seg_pos={0 0}, upload=478008]: Done begin upload segment. Dec 28 08:34:53.655162619 (545h5m53.526245664s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655175684 (545h5m53.526258729s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655926524 (545h5m53.527009569s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656153545 (545h5m53.527236591s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656339580 (545h5m53.527422615s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656657405 (545h5m53.52774045s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657123973 (545h5m53.528207019s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657446227 (545h5m53.528529262s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658059966 (545h5m53.529143001s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658479505 (545h5m53.52956254s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659150342 (545h5m53.530233387s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659482926 (545h5m53.530565971s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659605789 (545h5m53.530688834s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659895872 (545h5m53.530978907s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660306494 (545h5m53.531389529s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660662522 (545h5m53.531745567s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:55.372249885 (545h5m55.24333292s) [upload=478008]: Next returned nil segment Dec 28 08:34:55.372267248 (545h5m55.243350293s) [upload=478008]: Waiting for error group managing segments... Dec 28 08:34:55.659647340 (545h5m55.530730385s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.661694348 (545h5m55.532777383s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.663307369 (545h5m55.534390414s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.664639025 (545h5m55.53572206s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.671168919 (545h5m55.542251964s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.672574496 (545h5m55.543657531s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.673971776 (545h5m55.545054811s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680441596 (545h5m55.551524631s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680521839 (545h5m55.551604874s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.712067176 (545h5m55.583150221s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.940884955 (545h5m55.811967991s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.948922120 (545h5m55.820005155s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068098813 (545h5m55.939181848s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068123571 (545h5m55.939206616s) [seg_pos={0 0}, upload=478008]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:56.068349130 (545h5m55.939432175s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068358368 (545h5m55.939441403s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068646216 (545h5m55.939729261s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection Dec 28 08:34:56.068654592 (545h5m55.939737637s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068667766 (545h5m55.939750811s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068676033 (545h5m55.939759078s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068792815 (545h5m55.93987586s) [seg_pos={0 0}, upload=478008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:56.068880953 (545h5m55.939963988s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:56.068886644 (545h5m55.939969689s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:56.350764128 (545h5m56.221847183s) [upload=478008]: Done uploading segments... Dec 28 08:34:56.350785848 (545h5m56.221868883s) [upload=478008]: Upload finished. err: <nil> Duration: 6.152996814s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX...] duration: 5.001423686s trace: https://jaeger.dev.storj.io/trace/58488bbea5e068ff FAIL (2025-12-28 08:38:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001460326s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX..] duration: 3.724692377s trace: https://jaeger.dev.storj.io/trace/58488bbea5e068ff FAIL (2025-12-28 08:38:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.724724239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX....] duration: 7.198914975s trace: https://jaeger.dev.storj.io/trace/7aeea27d43bad8d4 FAIL (2025-12-28 08:38:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q) context deadline exceeded ======LOGS====== Dec 28 08:34:53.406660854 (545h5m53.277743889s) [upload=478008]: Starting upload Dec 28 08:34:53.406730828 (545h5m53.277813873s) [upload=478008]: Done starting upload Dec 28 08:34:53.406852530 (545h5m53.277935565s) [upload=478008]: Uploading segments... Dec 28 08:34:53.406999069 (545h5m53.278082114s) [upload=478008]: Got next segment. Inline: false Dec 28 08:34:53.407031791 (545h5m53.278114836s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:53.407060696 (545h5m53.278143741s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:53.655052049 (545h5m53.526135114s) [seg_pos={0 0}, upload=478008]: Begin upload segment... Dec 28 08:34:53.655107123 (545h5m53.526190168s) [seg_pos={0 0}, upload=478008]: Done begin upload segment. Dec 28 08:34:53.655162619 (545h5m53.526245664s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655175684 (545h5m53.526258729s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655926524 (545h5m53.527009569s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656153545 (545h5m53.527236591s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656339580 (545h5m53.527422615s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656657405 (545h5m53.52774045s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657123973 (545h5m53.528207019s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657446227 (545h5m53.528529262s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658059966 (545h5m53.529143001s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658479505 (545h5m53.52956254s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659150342 (545h5m53.530233387s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659482926 (545h5m53.530565971s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659605789 (545h5m53.530688834s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659895872 (545h5m53.530978907s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660306494 (545h5m53.531389529s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660662522 (545h5m53.531745567s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:55.372249885 (545h5m55.24333292s) [upload=478008]: Next returned nil segment Dec 28 08:34:55.372267248 (545h5m55.243350293s) [upload=478008]: Waiting for error group managing segments... Dec 28 08:34:55.659647340 (545h5m55.530730385s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.661694348 (545h5m55.532777383s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.663307369 (545h5m55.534390414s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.664639025 (545h5m55.53572206s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.671168919 (545h5m55.542251964s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.672574496 (545h5m55.543657531s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.673971776 (545h5m55.545054811s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680441596 (545h5m55.551524631s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680521839 (545h5m55.551604874s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.712067176 (545h5m55.583150221s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.940884955 (545h5m55.811967991s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.948922120 (545h5m55.820005155s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068098813 (545h5m55.939181848s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068123571 (545h5m55.939206616s) [seg_pos={0 0}, upload=478008]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:56.068349130 (545h5m55.939432175s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068358368 (545h5m55.939441403s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068646216 (545h5m55.939729261s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection Dec 28 08:34:56.068654592 (545h5m55.939737637s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068667766 (545h5m55.939750811s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068676033 (545h5m55.939759078s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068792815 (545h5m55.93987586s) [seg_pos={0 0}, upload=478008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:56.068880953 (545h5m55.939963988s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:56.068886644 (545h5m55.939969689s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:56.350764128 (545h5m56.221847183s) [upload=478008]: Done uploading segments... Dec 28 08:34:56.350785848 (545h5m56.221868883s) [upload=478008]: Upload finished. err: <nil> Duration: 7.198945113s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 14.828527564s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:38:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 14.828558933s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 10.890469732s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:37:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.89087859s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [......X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX.] duration: 3.586592009s trace: https://jaeger.dev.storj.io/trace/58488bbea5e068ff FAIL (2025-12-28 08:37:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.586671711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.036049502s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:37:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.036078086s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX...] duration: 5.883694165s trace: https://jaeger.dev.storj.io/trace/7aeea27d43bad8d4 FAIL (2025-12-28 08:37:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q) context deadline exceeded ======LOGS====== Dec 28 08:34:53.406660854 (545h5m53.277743889s) [upload=478008]: Starting upload Dec 28 08:34:53.406730828 (545h5m53.277813873s) [upload=478008]: Done starting upload Dec 28 08:34:53.406852530 (545h5m53.277935565s) [upload=478008]: Uploading segments... Dec 28 08:34:53.406999069 (545h5m53.278082114s) [upload=478008]: Got next segment. Inline: false Dec 28 08:34:53.407031791 (545h5m53.278114836s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:53.407060696 (545h5m53.278143741s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:53.655052049 (545h5m53.526135114s) [seg_pos={0 0}, upload=478008]: Begin upload segment... Dec 28 08:34:53.655107123 (545h5m53.526190168s) [seg_pos={0 0}, upload=478008]: Done begin upload segment. Dec 28 08:34:53.655162619 (545h5m53.526245664s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655175684 (545h5m53.526258729s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655926524 (545h5m53.527009569s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656153545 (545h5m53.527236591s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656339580 (545h5m53.527422615s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656657405 (545h5m53.52774045s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657123973 (545h5m53.528207019s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657446227 (545h5m53.528529262s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658059966 (545h5m53.529143001s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658479505 (545h5m53.52956254s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659150342 (545h5m53.530233387s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659482926 (545h5m53.530565971s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659605789 (545h5m53.530688834s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659895872 (545h5m53.530978907s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660306494 (545h5m53.531389529s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660662522 (545h5m53.531745567s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:55.372249885 (545h5m55.24333292s) [upload=478008]: Next returned nil segment Dec 28 08:34:55.372267248 (545h5m55.243350293s) [upload=478008]: Waiting for error group managing segments... Dec 28 08:34:55.659647340 (545h5m55.530730385s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.661694348 (545h5m55.532777383s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.663307369 (545h5m55.534390414s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.664639025 (545h5m55.53572206s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.671168919 (545h5m55.542251964s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.672574496 (545h5m55.543657531s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.673971776 (545h5m55.545054811s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680441596 (545h5m55.551524631s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680521839 (545h5m55.551604874s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.712067176 (545h5m55.583150221s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.940884955 (545h5m55.811967991s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.948922120 (545h5m55.820005155s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068098813 (545h5m55.939181848s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068123571 (545h5m55.939206616s) [seg_pos={0 0}, upload=478008]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:56.068349130 (545h5m55.939432175s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068358368 (545h5m55.939441403s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068646216 (545h5m55.939729261s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection Dec 28 08:34:56.068654592 (545h5m55.939737637s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068667766 (545h5m55.939750811s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068676033 (545h5m55.939759078s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068792815 (545h5m55.93987586s) [seg_pos={0 0}, upload=478008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:56.068880953 (545h5m55.939963988s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:56.068886644 (545h5m55.939969689s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:56.350764128 (545h5m56.221847183s) [upload=478008]: Done uploading segments... Dec 28 08:34:56.350785848 (545h5m56.221868883s) [upload=478008]: Upload finished. err: <nil> Duration: 5.883725374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX..] duration: 5.609326178s trace: https://jaeger.dev.storj.io/trace/7aeea27d43bad8d4 FAIL (2025-12-28 08:36:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q) context deadline exceeded ======LOGS====== Dec 28 08:34:53.406660854 (545h5m53.277743889s) [upload=478008]: Starting upload Dec 28 08:34:53.406730828 (545h5m53.277813873s) [upload=478008]: Done starting upload Dec 28 08:34:53.406852530 (545h5m53.277935565s) [upload=478008]: Uploading segments... Dec 28 08:34:53.406999069 (545h5m53.278082114s) [upload=478008]: Got next segment. Inline: false Dec 28 08:34:53.407031791 (545h5m53.278114836s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:53.407060696 (545h5m53.278143741s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:53.655052049 (545h5m53.526135114s) [seg_pos={0 0}, upload=478008]: Begin upload segment... Dec 28 08:34:53.655107123 (545h5m53.526190168s) [seg_pos={0 0}, upload=478008]: Done begin upload segment. Dec 28 08:34:53.655162619 (545h5m53.526245664s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655175684 (545h5m53.526258729s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655926524 (545h5m53.527009569s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656153545 (545h5m53.527236591s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656339580 (545h5m53.527422615s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656657405 (545h5m53.52774045s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657123973 (545h5m53.528207019s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657446227 (545h5m53.528529262s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658059966 (545h5m53.529143001s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658479505 (545h5m53.52956254s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659150342 (545h5m53.530233387s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659482926 (545h5m53.530565971s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659605789 (545h5m53.530688834s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659895872 (545h5m53.530978907s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660306494 (545h5m53.531389529s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660662522 (545h5m53.531745567s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:55.372249885 (545h5m55.24333292s) [upload=478008]: Next returned nil segment Dec 28 08:34:55.372267248 (545h5m55.243350293s) [upload=478008]: Waiting for error group managing segments... Dec 28 08:34:55.659647340 (545h5m55.530730385s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.661694348 (545h5m55.532777383s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.663307369 (545h5m55.534390414s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.664639025 (545h5m55.53572206s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.671168919 (545h5m55.542251964s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.672574496 (545h5m55.543657531s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.673971776 (545h5m55.545054811s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680441596 (545h5m55.551524631s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680521839 (545h5m55.551604874s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.712067176 (545h5m55.583150221s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.940884955 (545h5m55.811967991s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.948922120 (545h5m55.820005155s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068098813 (545h5m55.939181848s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068123571 (545h5m55.939206616s) [seg_pos={0 0}, upload=478008]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:56.068349130 (545h5m55.939432175s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068358368 (545h5m55.939441403s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068646216 (545h5m55.939729261s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection Dec 28 08:34:56.068654592 (545h5m55.939737637s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068667766 (545h5m55.939750811s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068676033 (545h5m55.939759078s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068792815 (545h5m55.93987586s) [seg_pos={0 0}, upload=478008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:56.068880953 (545h5m55.939963988s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:56.068886644 (545h5m55.939969689s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:56.350764128 (545h5m56.221847183s) [upload=478008]: Done uploading segments... Dec 28 08:34:56.350785848 (545h5m56.221868883s) [upload=478008]: Upload finished. err: <nil> Duration: 5.609357468s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.......X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX] duration: 5.001219478s trace: https://jaeger.dev.storj.io/trace/58488bbea5e068ff FAIL (2025-12-28 08:36:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001244416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 12.692376307s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:36:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.692396977s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX.X] duration: 5.001462632s trace: https://jaeger.dev.storj.io/trace/32869553e7e850ad FAIL (2025-12-28 08:36:01 UTC) orbiter warning: context deadline exceeded Duration: 5.001486668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.] duration: 5.915811412s trace: https://jaeger.dev.storj.io/trace/7aeea27d43bad8d4 FAIL (2025-12-28 08:35:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q) context deadline exceeded ======LOGS====== Dec 28 08:34:53.406660854 (545h5m53.277743889s) [upload=478008]: Starting upload Dec 28 08:34:53.406730828 (545h5m53.277813873s) [upload=478008]: Done starting upload Dec 28 08:34:53.406852530 (545h5m53.277935565s) [upload=478008]: Uploading segments... Dec 28 08:34:53.406999069 (545h5m53.278082114s) [upload=478008]: Got next segment. Inline: false Dec 28 08:34:53.407031791 (545h5m53.278114836s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:53.407060696 (545h5m53.278143741s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:53.655052049 (545h5m53.526135114s) [seg_pos={0 0}, upload=478008]: Begin upload segment... Dec 28 08:34:53.655107123 (545h5m53.526190168s) [seg_pos={0 0}, upload=478008]: Done begin upload segment. Dec 28 08:34:53.655162619 (545h5m53.526245664s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655175684 (545h5m53.526258729s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655926524 (545h5m53.527009569s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656153545 (545h5m53.527236591s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656339580 (545h5m53.527422615s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656657405 (545h5m53.52774045s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657123973 (545h5m53.528207019s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657446227 (545h5m53.528529262s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658059966 (545h5m53.529143001s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658479505 (545h5m53.52956254s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659150342 (545h5m53.530233387s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659482926 (545h5m53.530565971s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659605789 (545h5m53.530688834s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659895872 (545h5m53.530978907s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660306494 (545h5m53.531389529s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660662522 (545h5m53.531745567s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:55.372249885 (545h5m55.24333292s) [upload=478008]: Next returned nil segment Dec 28 08:34:55.372267248 (545h5m55.243350293s) [upload=478008]: Waiting for error group managing segments... Dec 28 08:34:55.659647340 (545h5m55.530730385s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.661694348 (545h5m55.532777383s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.663307369 (545h5m55.534390414s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.664639025 (545h5m55.53572206s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.671168919 (545h5m55.542251964s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.672574496 (545h5m55.543657531s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.673971776 (545h5m55.545054811s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680441596 (545h5m55.551524631s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680521839 (545h5m55.551604874s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.712067176 (545h5m55.583150221s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.940884955 (545h5m55.811967991s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.948922120 (545h5m55.820005155s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068098813 (545h5m55.939181848s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068123571 (545h5m55.939206616s) [seg_pos={0 0}, upload=478008]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:56.068349130 (545h5m55.939432175s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068358368 (545h5m55.939441403s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068646216 (545h5m55.939729261s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection Dec 28 08:34:56.068654592 (545h5m55.939737637s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068667766 (545h5m55.939750811s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068676033 (545h5m55.939759078s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068792815 (545h5m55.93987586s) [seg_pos={0 0}, upload=478008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:56.068880953 (545h5m55.939963988s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:56.068886644 (545h5m55.939969689s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:56.350764128 (545h5m56.221847183s) [upload=478008]: Done uploading segments... Dec 28 08:34:56.350785848 (545h5m56.221868883s) [upload=478008]: Upload finished. err: <nil> Duration: 5.915837041s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX.] duration: 3.411246972s trace: https://jaeger.dev.storj.io/trace/63b0ce8fc8fec88 FAIL (2025-12-28 08:35:38 UTC) orbiter warning: context deadline exceeded Duration: 3.41127195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.15209694s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:35:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.152123361s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX] duration: 30.003261298s trace: https://jaeger.dev.storj.io/trace/7aeea27d43bad8d4 FAIL (2025-12-28 08:35:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q) context deadline exceeded ======LOGS====== Dec 28 08:34:53.406660854 (545h5m53.277743889s) [upload=478008]: Starting upload Dec 28 08:34:53.406730828 (545h5m53.277813873s) [upload=478008]: Done starting upload Dec 28 08:34:53.406852530 (545h5m53.277935565s) [upload=478008]: Uploading segments... Dec 28 08:34:53.406999069 (545h5m53.278082114s) [upload=478008]: Got next segment. Inline: false Dec 28 08:34:53.407031791 (545h5m53.278114836s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:53.407060696 (545h5m53.278143741s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:53.655052049 (545h5m53.526135114s) [seg_pos={0 0}, upload=478008]: Begin upload segment... Dec 28 08:34:53.655107123 (545h5m53.526190168s) [seg_pos={0 0}, upload=478008]: Done begin upload segment. Dec 28 08:34:53.655162619 (545h5m53.526245664s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655175684 (545h5m53.526258729s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.655926524 (545h5m53.527009569s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656153545 (545h5m53.527236591s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656339580 (545h5m53.527422615s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.656657405 (545h5m53.52774045s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657123973 (545h5m53.528207019s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.657446227 (545h5m53.528529262s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658059966 (545h5m53.529143001s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.658479505 (545h5m53.52956254s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659150342 (545h5m53.530233387s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659482926 (545h5m53.530565971s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659605789 (545h5m53.530688834s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.659895872 (545h5m53.530978907s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660306494 (545h5m53.531389529s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:53.660662522 (545h5m53.531745567s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Uploading piece... Dec 28 08:34:55.372249885 (545h5m55.24333292s) [upload=478008]: Next returned nil segment Dec 28 08:34:55.372267248 (545h5m55.243350293s) [upload=478008]: Waiting for error group managing segments... Dec 28 08:34:55.659647340 (545h5m55.530730385s) [noise=true , address=145.79.227.51:4409 , piece_id=3THKXOECKRH3KTFIIYZMPTMNEVHZ4HIAJLFNAZOLEDQPVPZ2BY7Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.661694348 (545h5m55.532777383s) [noise=true , address=145.79.227.37:4567 , piece_id=CGCOITLZVMO26KPI7LZLUY52HDLDKVZ4RV7SQEFFA5RN723XGJMQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.663307369 (545h5m55.534390414s) [noise=true , address=145.79.227.39:4014 , piece_id=EEQFGVEGOSSBP26DZUHJA2OB5ZW3INABDITRZEGMY7TKVJC7X57Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.664639025 (545h5m55.53572206s) [noise=true , address=145.79.227.9:4365 , piece_id=TS7OFIFER3HBD3M5TTIPE4DYK7DFOLNKL27D5YGHHARWOWNNVMLQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.671168919 (545h5m55.542251964s) [noise=true , address=145.79.227.52:4447 , piece_id=NNDCX2BNZ3Z2QJSXNMWNVYXDWYP5CBAFQJWP4BC5ZVVFLX7IFZYQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.672574496 (545h5m55.543657531s) [noise=true , address=145.79.227.14:4837 , piece_id=2VZXQKLSALXD2U2P3IYCGX72RTZQBMFOQHAWCXZHOYOBMGTMJYCA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.673971776 (545h5m55.545054811s) [noise=true , address=145.79.227.29:4943 , piece_id=OQ7DZO566SH5DT32RCVYVFUJMFKEYC33XJ7TS5F3UAGPLD3UQI6A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680441596 (545h5m55.551524631s) [noise=true , address=145.79.227.42:4122 , piece_id=HBASEQHYS4WCPOODGBETBN5UNMCYYBUU2RNNSD3RTJIQRZP4GQNQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.680521839 (545h5m55.551604874s) [noise=true , address=145.79.227.45:4230 , piece_id=XHEFZEMJMJJZRB4WZUTC3XLZLDRFLD6LLJKKKHBH56I5MRYQBNUA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.712067176 (545h5m55.583150221s) [noise=true , address=145.79.227.19:4079 , piece_id=2PADUCFTJRYMC32KOGCZIO5M6VLJLRJSTTP2CS7A5ZFOJRLSRUCQ, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.940884955 (545h5m55.811967991s) [noise=true , address=45.91.116.9:4264 , piece_id=5CQDXE5D3HDL3BUMHWUOHSJBPZRCDDLQPHEUIIBNIMXQHRQWZW6Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:55.948922120 (545h5m55.820005155s) [noise=true , address=45.91.116.11:4119 , piece_id=DMZSK3AK2J2JSFO73VVO6A5A6OHLWPGWUO5UN3ODQYY5MJIW3HJA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068098813 (545h5m55.939181848s) [noise=true , address=145.79.227.10:4401 , piece_id=PAP2MARRTKSRWEVN45GUE67W5RUJUUL2YNMKU7JQYEIZ7DVHSB5A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: <nil> Dec 28 08:34:56.068123571 (545h5m55.939206616s) [seg_pos={0 0}, upload=478008]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:56.068349130 (545h5m55.939432175s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068358368 (545h5m55.939441403s) [noise=true , address=45.91.116.5:4300 , piece_id=D4P4KG5L5RVUYFW7WNEJFYU3DYWV55OGALCGSIE46KLQFVP5ZFFA, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068646216 (545h5m55.939729261s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51988->77.90.31.13:4629: use of closed network connection Dec 28 08:34:56.068654592 (545h5m55.939737637s) [noise=true , address=77.90.31.13:4629 , piece_id=W5MRWLQP7JBYYGSHMA5TMOKLKUXW2UKD6UVFTUOXJICJVLDB653Q, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068667766 (545h5m55.939750811s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:56.068676033 (545h5m55.939759078s) [noise=true , address=77.90.31.15:4700 , piece_id=VDUOU5SACSCWUHHK6SP4VCVJHHB6TAXSUDATMF5K2KQJPNTO6W3A, seg_pos={0 0}, upload=478008]: Long tail cancellation due to optimal threshold Dec 28 08:34:56.068792815 (545h5m55.93987586s) [seg_pos={0 0}, upload=478008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:56.068880953 (545h5m55.939963988s) [upload=478008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:56.068886644 (545h5m55.939969689s) [upload=478008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:56.350764128 (545h5m56.221847183s) [upload=478008]: Done uploading segments... Dec 28 08:34:56.350785848 (545h5m56.221868883s) [upload=478008]: Upload finished. err: <nil> Duration: 30.003275456s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXXX] duration: 5.001131191s trace: https://jaeger.dev.storj.io/trace/63b0ce8fc8fec88 FAIL (2025-12-28 08:34:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001151761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.X] duration: 30.002481482s trace: https://jaeger.dev.storj.io/trace/6a64357e4573c7dc FAIL (2025-12-28 08:34:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: I5BDPEW5KZOS5FGIDDCNNBARPA4LBMVRRGNJMSN4NRNTJ2YMDCVQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: BQELGBJMATJERNCQRBVLM6R25ILQB4QEOAAIVJJALXKYEZFF5KJQ) context deadline exceeded ======LOGS====== Dec 28 08:34:13.598564477 (545h5m13.469647522s) [upload=477997]: Starting upload Dec 28 08:34:13.598623439 (545h5m13.469706484s) [upload=477997]: Done starting upload Dec 28 08:34:13.598714422 (545h5m13.469797457s) [upload=477997]: Uploading segments... Dec 28 08:34:13.598819502 (545h5m13.469902537s) [upload=477997]: Got next segment. Inline: false Dec 28 08:34:13.598875218 (545h5m13.469958264s) [upload=477997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:34:13.598881540 (545h5m13.469964585s) [upload=477997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:34:13.839112525 (545h5m13.71019558s) [seg_pos={0 0}, upload=477997]: Begin upload segment... Dec 28 08:34:13.839157781 (545h5m13.710240816s) [seg_pos={0 0}, upload=477997]: Done begin upload segment. Dec 28 08:34:13.839182900 (545h5m13.710265945s) [noise=true , address=145.79.227.30:4761 , piece_id=PU3SRKQYDBUCE7JT6VAIM3V3RY5WRRGYR6XNRTDHIEMEBPGXDVBQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.839348113 (545h5m13.710431158s) [noise=true , address=45.91.116.14:4189 , piece_id=XRA6EDRR4Z5H7IBM7B5BOFXTZCLQMZIHKXB6SUL4WSVIQUKWOXLA, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.839743998 (545h5m13.710827043s) [noise=true , address=145.79.227.45:4229 , piece_id=Q53C2Z4QJJFGL7FX4Z4KZO7QDP5LXDFSYJU4T2CSTVIAHZBGS2DQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.840164719 (545h5m13.711247754s) [noise=true , address=145.79.227.19:4077 , piece_id=QBVVSJXJPMWYUEAUHU25ZG6RWMGCWVUYBWPAXXUMYEYHC2IPBYRQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.840428360 (545h5m13.711511395s) [noise=true , address=45.91.116.8:4045 , piece_id=BQELGBJMATJERNCQRBVLM6R25ILQB4QEOAAIVJJALXKYEZFF5KJQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.840561694 (545h5m13.711644739s) [noise=true , address=145.79.227.17:4970 , piece_id=RGCDFI4UY3ADRSODRS7B3727EOCVD5YXLKS44VWLCICVESAPMXOQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.840842339 (545h5m13.711925374s) [noise=true , address=145.79.227.6:4258 , piece_id=SVG2KBSYSMMOPX4GDZZ3JBU33SYF7S73DQY6K3ODTAM6HT7CQA2A, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.840944814 (545h5m13.712027859s) [noise=true , address=45.91.116.12:4227 , piece_id=I5BDPEW5KZOS5FGIDDCNNBARPA4LBMVRRGNJMSN4NRNTJ2YMDCVQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.841150305 (545h5m13.71223334s) [noise=true , address=45.91.116.4:4939 , piece_id=BZSERXNQR7MMPOD2C4IOML5C6A4BMHGENNMRTIEW24V5PQ34T7KQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.841442662 (545h5m13.712525697s) [noise=true , address=145.79.227.13:4804 , piece_id=BBVYUEBH6VOUTNAHFRNK2QC2WTK4PV3VZGF2KR4U7JMKWAJJZRJA, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.841486045 (545h5m13.71256908s) [noise=true , address=77.90.31.15:4697 , piece_id=TNGTBNQN7WHP33ZJJ3DHKA7ZUEI7IXH4KUYON565TUXASWS2YVNQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.841778241 (545h5m13.712861286s) [noise=true , address=145.79.227.42:4122 , piece_id=KVDZT5I6UTGLILO7XCYVC5WCV6LVKFTK2RYMG3SB7MUEKFW2XKCQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.842125432 (545h5m13.713208477s) [noise=true , address=145.79.227.28:4727 , piece_id=DV7A7EEZYOIDM3GJMC7ZURA3QH6KPJVWPC7TYUT3KPA5PQM3SORA, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.842300295 (545h5m13.71338333s) [noise=true , address=145.79.227.36:4517 , piece_id=CHIT6SAT6U5FAX44R73FEVPQE7X5DJ7PZWOXUT5AXQUJN3ED64FA, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.842528179 (545h5m13.713611224s) [noise=true , address=145.79.227.50:4374 , piece_id=A4EEZW6XTERQKYXP7MZKPLTGVUQSJ7PXMGKEKNKCSDGFVPN4VPHQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:13.842628991 (545h5m13.713712027s) [noise=true , address=145.79.227.11:4729 , piece_id=BOGEFBWUWOI5KRR56HYDRE2NV5ER7CVL45HA5W2TURVO5XTDZ4BQ, seg_pos={0 0}, upload=477997]: Uploading piece... Dec 28 08:34:15.463952415 (545h5m15.33503546s) [upload=477997]: Next returned nil segment Dec 28 08:34:15.463968887 (545h5m15.335051922s) [upload=477997]: Waiting for error group managing segments... Dec 28 08:34:15.840830943 (545h5m15.711913978s) [noise=true , address=145.79.227.6:4258 , piece_id=SVG2KBSYSMMOPX4GDZZ3JBU33SYF7S73DQY6K3ODTAM6HT7CQA2A, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.843072090 (545h5m15.714155125s) [noise=true , address=145.79.227.28:4727 , piece_id=DV7A7EEZYOIDM3GJMC7ZURA3QH6KPJVWPC7TYUT3KPA5PQM3SORA, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.847612696 (545h5m15.718695741s) [noise=true , address=145.79.227.11:4729 , piece_id=BOGEFBWUWOI5KRR56HYDRE2NV5ER7CVL45HA5W2TURVO5XTDZ4BQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.851043539 (545h5m15.722126574s) [noise=true , address=145.79.227.50:4374 , piece_id=A4EEZW6XTERQKYXP7MZKPLTGVUQSJ7PXMGKEKNKCSDGFVPN4VPHQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.852845210 (545h5m15.723928245s) [noise=true , address=145.79.227.13:4804 , piece_id=BBVYUEBH6VOUTNAHFRNK2QC2WTK4PV3VZGF2KR4U7JMKWAJJZRJA, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.858968280 (545h5m15.730051315s) [noise=true , address=145.79.227.36:4517 , piece_id=CHIT6SAT6U5FAX44R73FEVPQE7X5DJ7PZWOXUT5AXQUJN3ED64FA, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.859408287 (545h5m15.730491332s) [noise=true , address=145.79.227.17:4970 , piece_id=RGCDFI4UY3ADRSODRS7B3727EOCVD5YXLKS44VWLCICVESAPMXOQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.860055820 (545h5m15.731138855s) [noise=true , address=145.79.227.42:4122 , piece_id=KVDZT5I6UTGLILO7XCYVC5WCV6LVKFTK2RYMG3SB7MUEKFW2XKCQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.878766800 (545h5m15.749849835s) [noise=true , address=145.79.227.45:4229 , piece_id=Q53C2Z4QJJFGL7FX4Z4KZO7QDP5LXDFSYJU4T2CSTVIAHZBGS2DQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:15.884833041 (545h5m15.755916086s) [noise=true , address=145.79.227.19:4077 , piece_id=QBVVSJXJPMWYUEAUHU25ZG6RWMGCWVUYBWPAXXUMYEYHC2IPBYRQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:16.077857576 (545h5m15.948940621s) [noise=true , address=45.91.116.4:4939 , piece_id=BZSERXNQR7MMPOD2C4IOML5C6A4BMHGENNMRTIEW24V5PQ34T7KQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:16.167847404 (545h5m16.038930459s) [noise=true , address=45.91.116.12:4227 , piece_id=I5BDPEW5KZOS5FGIDDCNNBARPA4LBMVRRGNJMSN4NRNTJ2YMDCVQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:16.510032929 (545h5m16.381115964s) [noise=true , address=45.91.116.8:4045 , piece_id=BQELGBJMATJERNCQRBVLM6R25ILQB4QEOAAIVJJALXKYEZFF5KJQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: <nil> Dec 28 08:34:16.510062104 (545h5m16.381145149s) [seg_pos={0 0}, upload=477997]: Segment reached optimal threshold of 13 pieces. Dec 28 08:34:16.510340574 (545h5m16.381423639s) [noise=true , address=45.91.116.14:4189 , piece_id=XRA6EDRR4Z5H7IBM7B5BOFXTZCLQMZIHKXB6SUL4WSVIQUKWOXLA, seg_pos={0 0}, upload=477997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:16.510350975 (545h5m16.38143401s) [noise=true , address=45.91.116.14:4189 , piece_id=XRA6EDRR4Z5H7IBM7B5BOFXTZCLQMZIHKXB6SUL4WSVIQUKWOXLA, seg_pos={0 0}, upload=477997]: Long tail cancellation due to optimal threshold Dec 28 08:34:16.510401911 (545h5m16.381484946s) [noise=true , address=145.79.227.30:4761 , piece_id=PU3SRKQYDBUCE7JT6VAIM3V3RY5WRRGYR6XNRTDHIEMEBPGXDVBQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 08:34:16.510413033 (545h5m16.381496078s) [noise=true , address=145.79.227.30:4761 , piece_id=PU3SRKQYDBUCE7JT6VAIM3V3RY5WRRGYR6XNRTDHIEMEBPGXDVBQ, seg_pos={0 0}, upload=477997]: Long tail cancellation due to optimal threshold Dec 28 08:34:16.510675743 (545h5m16.381758788s) [noise=true , address=77.90.31.15:4697 , piece_id=TNGTBNQN7WHP33ZJJ3DHKA7ZUEI7IXH4KUYON565TUXASWS2YVNQ, seg_pos={0 0}, upload=477997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: context canceled; context canceled Dec 28 08:34:16.510685120 (545h5m16.381768155s) [noise=true , address=77.90.31.15:4697 , piece_id=TNGTBNQN7WHP33ZJJ3DHKA7ZUEI7IXH4KUYON565TUXASWS2YVNQ, seg_pos={0 0}, upload=477997]: Long tail cancellation due to optimal threshold Dec 28 08:34:16.510708726 (545h5m16.381791781s) [seg_pos={0 0}, upload=477997]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:34:16.510800571 (545h5m16.381883616s) [upload=477997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:34:16.510806812 (545h5m16.381889858s) [upload=477997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:34:16.806324564 (545h5m16.677407609s) [upload=477997]: Done uploading segments... Dec 28 08:34:16.806345655 (545h5m16.67742869s) [upload=477997]: Upload finished. err: <nil> Duration: 30.00250622s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 10.537836849s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:34:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.537866405s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XXX] duration: 5.000782198s trace: https://jaeger.dev.storj.io/trace/56410b139a7a5243 FAIL (2025-12-28 08:34:13 UTC) orbiter warning: context deadline exceeded Duration: 5.000811134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 17.318474404s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:34:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 17.318521204s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 14.157540625s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:33:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 14.157575962s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.] duration: 5.916351504s trace: https://jaeger.dev.storj.io/trace/4203743cdf15a1c1 FAIL (2025-12-28 08:33:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:32:09.018803043 (545h3m8.889886089s) [upload=477967]: Starting upload Dec 28 08:32:09.018890960 (545h3m8.889974005s) [upload=477967]: Done starting upload Dec 28 08:32:09.018996802 (545h3m8.890079857s) [upload=477967]: Uploading segments... Dec 28 08:32:09.019170372 (545h3m8.890253417s) [upload=477967]: Got next segment. Inline: false Dec 28 08:32:09.019209437 (545h3m8.890292482s) [upload=477967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:32:09.019217533 (545h3m8.890300588s) [upload=477967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:32:09.264081299 (545h3m9.135164334s) [seg_pos={0 0}, upload=477967]: Begin upload segment... Dec 28 08:32:09.264134430 (545h3m9.135217475s) [seg_pos={0 0}, upload=477967]: Done begin upload segment. Dec 28 08:32:09.264168275 (545h3m9.13525132s) [noise=true , address=45.91.116.6:4975 , piece_id=FHYEBD3TZEJT2V2NF5IHXJDYVJAMCNMDLKLXTDLFP27W5EMESD7A, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.264858920 (545h3m9.135941965s) [noise=true , address=145.79.227.11:4729 , piece_id=D2IHZZR7D5YYMWBWKR22GIZZLJVVPDSM22HXUORVBOEH2YELJCQQ, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.265230948 (545h3m9.136313993s) [noise=true , address=145.79.227.35:4483 , piece_id=OJO2HAKASBOJNU6ZJ2IHL6JR22U3OCCXZNZXWPANC2KFCOSW5MCQ, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.265504749 (545h3m9.136587794s) [noise=true , address=145.79.227.39:4015 , piece_id=MVQ5GGGPD4DQWJFFSFTPAF7TDYDX2T4FRCXANCJTUSUGLCALPO6A, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.265971879 (545h3m9.137054914s) [noise=true , address=145.79.227.36:4518 , piece_id=L7EAQSQK6BEDQPRN3ZXO4YD7AZENAHH7NAAMPXHBMPSCKXS4QVTQ, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.266257823 (545h3m9.137340859s) [noise=true , address=77.90.31.12:4590 , piece_id=36Q2U72JAOLPVUKVROYVQ7KYU5ENRN7XQOERGQVFTFRAKISTUOHA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.266528770 (545h3m9.137611815s) [noise=true , address=145.79.227.31:4797 , piece_id=JB54UQEAISMGHXITWFMREWBFZXI5S6KKBP73QUWR3Y2F4HVER45Q, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.266806829 (545h3m9.137889864s) [noise=true , address=145.79.227.2:4114 , piece_id=U5A2E42737PXOJBESU4TDDF7RODNYJW65T5RA6VCPIO22I6CK23Q, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.267204666 (545h3m9.138287701s) [noise=true , address=145.79.227.12:4767 , piece_id=YHOEDT3TPV4MEDSS6HWPBJ63WPQMEV6FOKA5BOSGUHRQ23UHU2OA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.267572417 (545h3m9.138655462s) [noise=true , address=145.79.227.46:4603 , piece_id=RPBUTFBL6J5ZHWB4VKYUM2JBS43Z6WIOBR7LZDDC7XTG2FVNMWNA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.268830993 (545h3m9.139914028s) [noise=true , address=145.79.227.45:4229 , piece_id=IJ3Q6LH7T6S7UWQ4DCPY5IFN3KBX6MGAPPO52OM46WYB7S2WAVQA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.271203611 (545h3m9.142286656s) [noise=true , address=145.79.227.16:4909 , piece_id=ZAMZSRMWSFAVOI2FMX4AFFV5G4TNHW3BZWXQKF4KBRBDW4XHQRAA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.272936771 (545h3m9.144019807s) [noise=true , address=45.91.116.9:4264 , piece_id=QO2PEJDFOMW7UKEYWPE7UEDFKZHD5U5L22RTQT3ZH5MACDYN3X4Q, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.273339379 (545h3m9.144422424s) [noise=true , address=45.91.116.10:4082 , piece_id=6GOQBK4WDG7TDGCEJ4O5XQSN65W5Q4DJJ7EIXHE5HR43YR6WAOEA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.276584287 (545h3m9.147667322s) [noise=true , address=45.91.116.14:4189 , piece_id=2Z66M22SIEDYFY5OLSW4PNUMTIBLBPGHJATGDWFFCD6F32LM5RIA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.283564800 (545h3m9.154647845s) [noise=true , address=45.91.116.5:4297 , piece_id=2UJBY6GBNAG7CET4K7XZMN62ZDDOPRDB5DPXT6WALUW7V4S46HSQ, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:10.889157598 (545h3m10.760240633s) [upload=477967]: Next returned nil segment Dec 28 08:32:10.889175903 (545h3m10.760258939s) [upload=477967]: Waiting for error group managing segments... Dec 28 08:32:11.274061237 (545h3m11.145144282s) [noise=true , address=145.79.227.46:4603 , piece_id=RPBUTFBL6J5ZHWB4VKYUM2JBS43Z6WIOBR7LZDDC7XTG2FVNMWNA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.274993943 (545h3m11.146076998s) [noise=true , address=145.79.227.31:4797 , piece_id=JB54UQEAISMGHXITWFMREWBFZXI5S6KKBP73QUWR3Y2F4HVER45Q, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.275899578 (545h3m11.146982623s) [noise=true , address=145.79.227.36:4518 , piece_id=L7EAQSQK6BEDQPRN3ZXO4YD7AZENAHH7NAAMPXHBMPSCKXS4QVTQ, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.276890134 (545h3m11.14797318s) [noise=true , address=145.79.227.2:4114 , piece_id=U5A2E42737PXOJBESU4TDDF7RODNYJW65T5RA6VCPIO22I6CK23Q, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.277297289 (545h3m11.148380344s) [noise=true , address=145.79.227.11:4729 , piece_id=D2IHZZR7D5YYMWBWKR22GIZZLJVVPDSM22HXUORVBOEH2YELJCQQ, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.278127881 (545h3m11.149210926s) [noise=true , address=145.79.227.39:4015 , piece_id=MVQ5GGGPD4DQWJFFSFTPAF7TDYDX2T4FRCXANCJTUSUGLCALPO6A, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.284751463 (545h3m11.155834508s) [noise=true , address=145.79.227.16:4909 , piece_id=ZAMZSRMWSFAVOI2FMX4AFFV5G4TNHW3BZWXQKF4KBRBDW4XHQRAA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.284771912 (545h3m11.155854947s) [noise=true , address=145.79.227.45:4229 , piece_id=IJ3Q6LH7T6S7UWQ4DCPY5IFN3KBX6MGAPPO52OM46WYB7S2WAVQA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.286616875 (545h3m11.157699921s) [noise=true , address=145.79.227.12:4767 , piece_id=YHOEDT3TPV4MEDSS6HWPBJ63WPQMEV6FOKA5BOSGUHRQ23UHU2OA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.438106540 (545h3m11.309189585s) [noise=true , address=77.90.31.12:4590 , piece_id=36Q2U72JAOLPVUKVROYVQ7KYU5ENRN7XQOERGQVFTFRAKISTUOHA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.496137592 (545h3m11.367220637s) [noise=true , address=45.91.116.14:4189 , piece_id=2Z66M22SIEDYFY5OLSW4PNUMTIBLBPGHJATGDWFFCD6F32LM5RIA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.551440139 (545h3m11.422523174s) [noise=true , address=45.91.116.9:4264 , piece_id=QO2PEJDFOMW7UKEYWPE7UEDFKZHD5U5L22RTQT3ZH5MACDYN3X4Q, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.555557369 (545h3m11.426640404s) [noise=true , address=45.91.116.6:4975 , piece_id=FHYEBD3TZEJT2V2NF5IHXJDYVJAMCNMDLKLXTDLFP27W5EMESD7A, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.555576476 (545h3m11.426659511s) [seg_pos={0 0}, upload=477967]: Segment reached optimal threshold of 13 pieces. Dec 28 08:32:11.555864544 (545h3m11.426947589s) [noise=true , address=45.91.116.10:4082 , piece_id=6GOQBK4WDG7TDGCEJ4O5XQSN65W5Q4DJJ7EIXHE5HR43YR6WAOEA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 08:32:11.555873571 (545h3m11.426956616s) [noise=true , address=45.91.116.10:4082 , piece_id=6GOQBK4WDG7TDGCEJ4O5XQSN65W5Q4DJJ7EIXHE5HR43YR6WAOEA, seg_pos={0 0}, upload=477967]: Long tail cancellation due to optimal threshold Dec 28 08:32:11.556093911 (545h3m11.427176946s) [noise=true , address=45.91.116.5:4297 , piece_id=2UJBY6GBNAG7CET4K7XZMN62ZDDOPRDB5DPXT6WALUW7V4S46HSQ, seg_pos={0 0}, upload=477967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:32:11.556101766 (545h3m11.427184811s) [noise=true , address=45.91.116.5:4297 , piece_id=2UJBY6GBNAG7CET4K7XZMN62ZDDOPRDB5DPXT6WALUW7V4S46HSQ, seg_pos={0 0}, upload=477967]: Long tail cancellation due to optimal threshold Dec 28 08:32:11.556249677 (545h3m11.427332722s) [noise=true , address=145.79.227.35:4483 , piece_id=OJO2HAKASBOJNU6ZJ2IHL6JR22U3OCCXZNZXWPANC2KFCOSW5MCQ, seg_pos={0 0}, upload=477967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 08:32:11.556270407 (545h3m11.427353442s) [noise=true , address=145.79.227.35:4483 , piece_id=OJO2HAKASBOJNU6ZJ2IHL6JR22U3OCCXZNZXWPANC2KFCOSW5MCQ, seg_pos={0 0}, upload=477967]: Long tail cancellation due to optimal threshold Dec 28 08:32:11.556328206 (545h3m11.427411242s) [seg_pos={0 0}, upload=477967]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:32:11.556423729 (545h3m11.427506774s) [upload=477967]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:32:11.556429059 (545h3m11.427512094s) [upload=477967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:32:11.842575796 (545h3m11.713658831s) [upload=477967]: Done uploading segments... Dec 28 08:32:11.842594292 (545h3m11.713677327s) [upload=477967]: Upload finished. err: <nil> Duration: 5.916380548s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....XX] duration: 5.001253297s trace: https://jaeger.dev.storj.io/trace/5f4b5bf2ede3a97c FAIL (2025-12-28 08:33:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001279556s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 10.724077267s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:33:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.724121122s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....X] duration: 5.000662652s trace: https://jaeger.dev.storj.io/trace/1e5590b754f07d62 FAIL (2025-12-28 08:32:52 UTC) orbiter warning: context deadline exceeded Duration: 5.000687308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX] duration: 30.002392328s trace: https://jaeger.dev.storj.io/trace/4203743cdf15a1c1 FAIL (2025-12-28 08:32:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:32:09.018803043 (545h3m8.889886089s) [upload=477967]: Starting upload Dec 28 08:32:09.018890960 (545h3m8.889974005s) [upload=477967]: Done starting upload Dec 28 08:32:09.018996802 (545h3m8.890079857s) [upload=477967]: Uploading segments... Dec 28 08:32:09.019170372 (545h3m8.890253417s) [upload=477967]: Got next segment. Inline: false Dec 28 08:32:09.019209437 (545h3m8.890292482s) [upload=477967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:32:09.019217533 (545h3m8.890300588s) [upload=477967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:32:09.264081299 (545h3m9.135164334s) [seg_pos={0 0}, upload=477967]: Begin upload segment... Dec 28 08:32:09.264134430 (545h3m9.135217475s) [seg_pos={0 0}, upload=477967]: Done begin upload segment. Dec 28 08:32:09.264168275 (545h3m9.13525132s) [noise=true , address=45.91.116.6:4975 , piece_id=FHYEBD3TZEJT2V2NF5IHXJDYVJAMCNMDLKLXTDLFP27W5EMESD7A, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.264858920 (545h3m9.135941965s) [noise=true , address=145.79.227.11:4729 , piece_id=D2IHZZR7D5YYMWBWKR22GIZZLJVVPDSM22HXUORVBOEH2YELJCQQ, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.265230948 (545h3m9.136313993s) [noise=true , address=145.79.227.35:4483 , piece_id=OJO2HAKASBOJNU6ZJ2IHL6JR22U3OCCXZNZXWPANC2KFCOSW5MCQ, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.265504749 (545h3m9.136587794s) [noise=true , address=145.79.227.39:4015 , piece_id=MVQ5GGGPD4DQWJFFSFTPAF7TDYDX2T4FRCXANCJTUSUGLCALPO6A, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.265971879 (545h3m9.137054914s) [noise=true , address=145.79.227.36:4518 , piece_id=L7EAQSQK6BEDQPRN3ZXO4YD7AZENAHH7NAAMPXHBMPSCKXS4QVTQ, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.266257823 (545h3m9.137340859s) [noise=true , address=77.90.31.12:4590 , piece_id=36Q2U72JAOLPVUKVROYVQ7KYU5ENRN7XQOERGQVFTFRAKISTUOHA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.266528770 (545h3m9.137611815s) [noise=true , address=145.79.227.31:4797 , piece_id=JB54UQEAISMGHXITWFMREWBFZXI5S6KKBP73QUWR3Y2F4HVER45Q, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.266806829 (545h3m9.137889864s) [noise=true , address=145.79.227.2:4114 , piece_id=U5A2E42737PXOJBESU4TDDF7RODNYJW65T5RA6VCPIO22I6CK23Q, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.267204666 (545h3m9.138287701s) [noise=true , address=145.79.227.12:4767 , piece_id=YHOEDT3TPV4MEDSS6HWPBJ63WPQMEV6FOKA5BOSGUHRQ23UHU2OA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.267572417 (545h3m9.138655462s) [noise=true , address=145.79.227.46:4603 , piece_id=RPBUTFBL6J5ZHWB4VKYUM2JBS43Z6WIOBR7LZDDC7XTG2FVNMWNA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.268830993 (545h3m9.139914028s) [noise=true , address=145.79.227.45:4229 , piece_id=IJ3Q6LH7T6S7UWQ4DCPY5IFN3KBX6MGAPPO52OM46WYB7S2WAVQA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.271203611 (545h3m9.142286656s) [noise=true , address=145.79.227.16:4909 , piece_id=ZAMZSRMWSFAVOI2FMX4AFFV5G4TNHW3BZWXQKF4KBRBDW4XHQRAA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.272936771 (545h3m9.144019807s) [noise=true , address=45.91.116.9:4264 , piece_id=QO2PEJDFOMW7UKEYWPE7UEDFKZHD5U5L22RTQT3ZH5MACDYN3X4Q, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.273339379 (545h3m9.144422424s) [noise=true , address=45.91.116.10:4082 , piece_id=6GOQBK4WDG7TDGCEJ4O5XQSN65W5Q4DJJ7EIXHE5HR43YR6WAOEA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.276584287 (545h3m9.147667322s) [noise=true , address=45.91.116.14:4189 , piece_id=2Z66M22SIEDYFY5OLSW4PNUMTIBLBPGHJATGDWFFCD6F32LM5RIA, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:09.283564800 (545h3m9.154647845s) [noise=true , address=45.91.116.5:4297 , piece_id=2UJBY6GBNAG7CET4K7XZMN62ZDDOPRDB5DPXT6WALUW7V4S46HSQ, seg_pos={0 0}, upload=477967]: Uploading piece... Dec 28 08:32:10.889157598 (545h3m10.760240633s) [upload=477967]: Next returned nil segment Dec 28 08:32:10.889175903 (545h3m10.760258939s) [upload=477967]: Waiting for error group managing segments... Dec 28 08:32:11.274061237 (545h3m11.145144282s) [noise=true , address=145.79.227.46:4603 , piece_id=RPBUTFBL6J5ZHWB4VKYUM2JBS43Z6WIOBR7LZDDC7XTG2FVNMWNA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.274993943 (545h3m11.146076998s) [noise=true , address=145.79.227.31:4797 , piece_id=JB54UQEAISMGHXITWFMREWBFZXI5S6KKBP73QUWR3Y2F4HVER45Q, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.275899578 (545h3m11.146982623s) [noise=true , address=145.79.227.36:4518 , piece_id=L7EAQSQK6BEDQPRN3ZXO4YD7AZENAHH7NAAMPXHBMPSCKXS4QVTQ, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.276890134 (545h3m11.14797318s) [noise=true , address=145.79.227.2:4114 , piece_id=U5A2E42737PXOJBESU4TDDF7RODNYJW65T5RA6VCPIO22I6CK23Q, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.277297289 (545h3m11.148380344s) [noise=true , address=145.79.227.11:4729 , piece_id=D2IHZZR7D5YYMWBWKR22GIZZLJVVPDSM22HXUORVBOEH2YELJCQQ, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.278127881 (545h3m11.149210926s) [noise=true , address=145.79.227.39:4015 , piece_id=MVQ5GGGPD4DQWJFFSFTPAF7TDYDX2T4FRCXANCJTUSUGLCALPO6A, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.284751463 (545h3m11.155834508s) [noise=true , address=145.79.227.16:4909 , piece_id=ZAMZSRMWSFAVOI2FMX4AFFV5G4TNHW3BZWXQKF4KBRBDW4XHQRAA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.284771912 (545h3m11.155854947s) [noise=true , address=145.79.227.45:4229 , piece_id=IJ3Q6LH7T6S7UWQ4DCPY5IFN3KBX6MGAPPO52OM46WYB7S2WAVQA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.286616875 (545h3m11.157699921s) [noise=true , address=145.79.227.12:4767 , piece_id=YHOEDT3TPV4MEDSS6HWPBJ63WPQMEV6FOKA5BOSGUHRQ23UHU2OA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.438106540 (545h3m11.309189585s) [noise=true , address=77.90.31.12:4590 , piece_id=36Q2U72JAOLPVUKVROYVQ7KYU5ENRN7XQOERGQVFTFRAKISTUOHA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.496137592 (545h3m11.367220637s) [noise=true , address=45.91.116.14:4189 , piece_id=2Z66M22SIEDYFY5OLSW4PNUMTIBLBPGHJATGDWFFCD6F32LM5RIA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.551440139 (545h3m11.422523174s) [noise=true , address=45.91.116.9:4264 , piece_id=QO2PEJDFOMW7UKEYWPE7UEDFKZHD5U5L22RTQT3ZH5MACDYN3X4Q, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.555557369 (545h3m11.426640404s) [noise=true , address=45.91.116.6:4975 , piece_id=FHYEBD3TZEJT2V2NF5IHXJDYVJAMCNMDLKLXTDLFP27W5EMESD7A, seg_pos={0 0}, upload=477967]: Done uploading piece. err: <nil> Dec 28 08:32:11.555576476 (545h3m11.426659511s) [seg_pos={0 0}, upload=477967]: Segment reached optimal threshold of 13 pieces. Dec 28 08:32:11.555864544 (545h3m11.426947589s) [noise=true , address=45.91.116.10:4082 , piece_id=6GOQBK4WDG7TDGCEJ4O5XQSN65W5Q4DJJ7EIXHE5HR43YR6WAOEA, seg_pos={0 0}, upload=477967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 08:32:11.555873571 (545h3m11.426956616s) [noise=true , address=45.91.116.10:4082 , piece_id=6GOQBK4WDG7TDGCEJ4O5XQSN65W5Q4DJJ7EIXHE5HR43YR6WAOEA, seg_pos={0 0}, upload=477967]: Long tail cancellation due to optimal threshold Dec 28 08:32:11.556093911 (545h3m11.427176946s) [noise=true , address=45.91.116.5:4297 , piece_id=2UJBY6GBNAG7CET4K7XZMN62ZDDOPRDB5DPXT6WALUW7V4S46HSQ, seg_pos={0 0}, upload=477967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:32:11.556101766 (545h3m11.427184811s) [noise=true , address=45.91.116.5:4297 , piece_id=2UJBY6GBNAG7CET4K7XZMN62ZDDOPRDB5DPXT6WALUW7V4S46HSQ, seg_pos={0 0}, upload=477967]: Long tail cancellation due to optimal threshold Dec 28 08:32:11.556249677 (545h3m11.427332722s) [noise=true , address=145.79.227.35:4483 , piece_id=OJO2HAKASBOJNU6ZJ2IHL6JR22U3OCCXZNZXWPANC2KFCOSW5MCQ, seg_pos={0 0}, upload=477967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 08:32:11.556270407 (545h3m11.427353442s) [noise=true , address=145.79.227.35:4483 , piece_id=OJO2HAKASBOJNU6ZJ2IHL6JR22U3OCCXZNZXWPANC2KFCOSW5MCQ, seg_pos={0 0}, upload=477967]: Long tail cancellation due to optimal threshold Dec 28 08:32:11.556328206 (545h3m11.427411242s) [seg_pos={0 0}, upload=477967]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:32:11.556423729 (545h3m11.427506774s) [upload=477967]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:32:11.556429059 (545h3m11.427512094s) [upload=477967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:32:11.842575796 (545h3m11.713658831s) [upload=477967]: Done uploading segments... Dec 28 08:32:11.842594292 (545h3m11.713677327s) [upload=477967]: Upload finished. err: <nil> Duration: 30.002416875s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 8.856875263s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:32:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.856900079s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [....XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X....] duration: 3.54690395s trace: https://jaeger.dev.storj.io/trace/6fd820859e8b0174 FAIL (2025-12-28 08:32:02 UTC) orbiter warning: context deadline exceeded Duration: 3.54693528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.....XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X...] duration: 3.40634415s trace: https://jaeger.dev.storj.io/trace/6fd820859e8b0174 FAIL (2025-12-28 08:31:46 UTC) orbiter warning: context deadline exceeded Duration: 3.40637085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.X] duration: 30.003021898s trace: https://jaeger.dev.storj.io/trace/6327dc049639a824 FAIL (2025-12-28 08:31:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: NYQX5U2VIBTLPH56GQ5DIWAGJTYXRI5A5HQEUVTKV6M4RHRFVLKA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: ULP5UXUN36HUCK2ODVLMPVCDK2P4MJGQ3S6OXHPOVWWWVZUQRWRQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: AFHB5X7XUT5QQQGMY6NRGIV4WEXD3AHEI6SHZV4FY7ZQNQASZLHQ) context deadline exceeded ======LOGS====== 1m55.882792047s) [upload=477953]: Uploading segments... Dec 28 08:30:56.011871191 (545h1m55.882954246s) [upload=477953]: Got next segment. Inline: false Dec 28 08:30:56.011918260 (545h1m55.883001305s) [upload=477953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:30:56.011924842 (545h1m55.883007888s) [upload=477953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:30:56.255682964 (545h1m56.126766009s) [seg_pos={0 0}, upload=477953]: Begin upload segment... Dec 28 08:30:56.255751284 (545h1m56.126834329s) [seg_pos={0 0}, upload=477953]: Done begin upload segment. Dec 28 08:30:56.255808242 (545h1m56.126891287s) [noise=true , address=145.79.227.12:4765 , piece_id=HSCEKK73LDWWYCP2JDHVP7VRHCZBG2AJXZA6PFMRKJRRSZYDHQSQ, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.255820506 (545h1m56.126903551s) [noise=true , address=145.79.227.42:4122 , piece_id=6QWYRRY45BAUBCFWU5TNMAY5YZS2NIMYHRAZSFL2WQL22A3VGO2A, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.256487195 (545h1m56.12757024s) [noise=true , address=45.91.116.8:4048 , piece_id=ULP5UXUN36HUCK2ODVLMPVCDK2P4MJGQ3S6OXHPOVWWWVZUQRWRQ, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.256522422 (545h1m56.127605467s) [noise=true , address=145.79.227.46:4601 , piece_id=HD4W2UGR6SLC5XWLMDXAXACRQHEWGBYJ3LBFOIUSATBN4JRGHBLA, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.256956498 (545h1m56.128039533s) [noise=true , address=145.79.227.24:4581 , piece_id=GQC4BJHIG3XTE2VHF7LHUMZWDK5W2W5S6ZL5II3S5GATSWV2HLHQ, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.256987297 (545h1m56.128070332s) [noise=true , address=45.91.116.12:4226 , piece_id=4DRUZF7FD5HSPCN6IVZJI3OWGL2BPASVLYUMVULKK5D6CN27YG4Q, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.257353765 (545h1m56.12843681s) [noise=true , address=77.90.31.15:4697 , piece_id=AFHB5X7XUT5QQQGMY6NRGIV4WEXD3AHEI6SHZV4FY7ZQNQASZLHQ, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.257499472 (545h1m56.128582517s) [noise=true , address=77.90.31.14:4664 , piece_id=IYVLLKUU4GG5NX574QG3YTMRAIPA5UTELJFM5L55WSCAJO2DXDYA, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.257748917 (545h1m56.128831962s) [noise=true , address=145.79.227.52:4447 , piece_id=NKZDG4AADE76LD3IBXEI3LVO6IG6VDTPG2Y36MJCCGR7M4GXXQSA, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.258101458 (545h1m56.129184503s) [noise=true , address=145.79.227.18:4007 , piece_id=A2PZJPA6BNMTPS45K4W7GCS55GJCEGUW2QGQZ5F4V2POFLJ5FRFA, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.258473928 (545h1m56.129556963s) [noise=true , address=145.79.227.17:4972 , piece_id=HMPDIKYDFDBUHPYBOS7ZHR6QNFPTI4PTGNMJVUBODSKZ47KUM7OA, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.258598184 (545h1m56.129681229s) [noise=true , address=145.79.227.44:4195 , piece_id=XBJLDVEQX7LKVXRGWXQTOW3TP7KEY4QQ3U6JZQXPDCB6OMZMMTQA, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.258868900 (545h1m56.129951935s) [noise=true , address=145.79.227.27:4689 , piece_id=M63H4WQFQLRARMA6NXS4WFIK5N3W3B7RTQEFFU2E3TK75ZO6UBNQ, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.259299801 (545h1m56.130382846s) [noise=true , address=145.79.227.16:4911 , piece_id=727JG2LA3ANR6F374ODSMATVMGUT3RW3CKEGMP63MZWP646HPLXA, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.259337903 (545h1m56.130420948s) [noise=true , address=77.90.31.12:4590 , piece_id=NYQX5U2VIBTLPH56GQ5DIWAGJTYXRI5A5HQEUVTKV6M4RHRFVLKA, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:56.259683211 (545h1m56.130766246s) [noise=true , address=145.79.227.36:4519 , piece_id=7TFKDGGDUSIMEYBUPNF4Q3IAZHJE7RGUGRGPGXOBJ2ATOCBX7Y2A, seg_pos={0 0}, upload=477953]: Uploading piece... Dec 28 08:30:57.863891233 (545h1m57.734974278s) [upload=477953]: Next returned nil segment Dec 28 08:30:57.863906002 (545h1m57.734989037s) [upload=477953]: Waiting for error group managing segments... Dec 28 08:30:58.251558335 (545h1m58.12264137s) [noise=true , address=145.79.227.12:4765 , piece_id=HSCEKK73LDWWYCP2JDHVP7VRHCZBG2AJXZA6PFMRKJRRSZYDHQSQ, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.260206211 (545h1m58.131289256s) [noise=true , address=145.79.227.36:4519 , piece_id=7TFKDGGDUSIMEYBUPNF4Q3IAZHJE7RGUGRGPGXOBJ2ATOCBX7Y2A, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.260692958 (545h1m58.131775993s) [noise=true , address=145.79.227.52:4447 , piece_id=NKZDG4AADE76LD3IBXEI3LVO6IG6VDTPG2Y36MJCCGR7M4GXXQSA, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.266227878 (545h1m58.137310913s) [noise=true , address=145.79.227.18:4007 , piece_id=A2PZJPA6BNMTPS45K4W7GCS55GJCEGUW2QGQZ5F4V2POFLJ5FRFA, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.267208675 (545h1m58.13829171s) [noise=true , address=145.79.227.24:4581 , piece_id=GQC4BJHIG3XTE2VHF7LHUMZWDK5W2W5S6ZL5II3S5GATSWV2HLHQ, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.267978472 (545h1m58.139061517s) [noise=true , address=145.79.227.42:4122 , piece_id=6QWYRRY45BAUBCFWU5TNMAY5YZS2NIMYHRAZSFL2WQL22A3VGO2A, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.270955129 (545h1m58.142038164s) [noise=true , address=145.79.227.27:4689 , piece_id=M63H4WQFQLRARMA6NXS4WFIK5N3W3B7RTQEFFU2E3TK75ZO6UBNQ, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.271379798 (545h1m58.142462833s) [noise=true , address=145.79.227.16:4911 , piece_id=727JG2LA3ANR6F374ODSMATVMGUT3RW3CKEGMP63MZWP646HPLXA, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.284487658 (545h1m58.155570703s) [noise=true , address=145.79.227.46:4601 , piece_id=HD4W2UGR6SLC5XWLMDXAXACRQHEWGBYJ3LBFOIUSATBN4JRGHBLA, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.294410954 (545h1m58.165493999s) [noise=true , address=145.79.227.17:4972 , piece_id=HMPDIKYDFDBUHPYBOS7ZHR6QNFPTI4PTGNMJVUBODSKZ47KUM7OA, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.528514441 (545h1m58.399597486s) [noise=true , address=77.90.31.12:4590 , piece_id=NYQX5U2VIBTLPH56GQ5DIWAGJTYXRI5A5HQEUVTKV6M4RHRFVLKA, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.664765619 (545h1m58.535848664s) [noise=true , address=45.91.116.8:4048 , piece_id=ULP5UXUN36HUCK2ODVLMPVCDK2P4MJGQ3S6OXHPOVWWWVZUQRWRQ, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.683085684 (545h1m58.554168719s) [noise=true , address=77.90.31.15:4697 , piece_id=AFHB5X7XUT5QQQGMY6NRGIV4WEXD3AHEI6SHZV4FY7ZQNQASZLHQ, seg_pos={0 0}, upload=477953]: Done uploading piece. err: <nil> Dec 28 08:30:58.683105060 (545h1m58.554188095s) [seg_pos={0 0}, upload=477953]: Segment reached optimal threshold of 13 pieces. Dec 28 08:30:58.683335660 (545h1m58.554418695s) [noise=true , address=145.79.227.44:4195 , piece_id=XBJLDVEQX7LKVXRGWXQTOW3TP7KEY4QQ3U6JZQXPDCB6OMZMMTQA, seg_pos={0 0}, upload=477953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 08:30:58.683342813 (545h1m58.554425848s) [noise=true , address=145.79.227.44:4195 , piece_id=XBJLDVEQX7LKVXRGWXQTOW3TP7KEY4QQ3U6JZQXPDCB6OMZMMTQA, seg_pos={0 0}, upload=477953]: Long tail cancellation due to optimal threshold Dec 28 08:30:58.683524880 (545h1m58.554607935s) [noise=true , address=77.90.31.14:4664 , piece_id=IYVLLKUU4GG5NX574QG3YTMRAIPA5UTELJFM5L55WSCAJO2DXDYA, seg_pos={0 0}, upload=477953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:43708->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:43708->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43708->77.90.31.14:4664: use of closed network connection Dec 28 08:30:58.683558514 (545h1m58.554641559s) [noise=true , address=77.90.31.14:4664 , piece_id=IYVLLKUU4GG5NX574QG3YTMRAIPA5UTELJFM5L55WSCAJO2DXDYA, seg_pos={0 0}, upload=477953]: Long tail cancellation due to optimal threshold Dec 28 08:30:58.683726113 (545h1m58.554809158s) [noise=true , address=45.91.116.12:4226 , piece_id=4DRUZF7FD5HSPCN6IVZJI3OWGL2BPASVLYUMVULKK5D6CN27YG4Q, seg_pos={0 0}, upload=477953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42220->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42220->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42220->45.91.116.12:4226: use of closed network connection Dec 28 08:30:58.683736052 (545h1m58.554819097s) [noise=true , address=45.91.116.12:4226 , piece_id=4DRUZF7FD5HSPCN6IVZJI3OWGL2BPASVLYUMVULKK5D6CN27YG4Q, seg_pos={0 0}, upload=477953]: Long tail cancellation due to optimal threshold Dec 28 08:30:58.683768964 (545h1m58.554851999s) [seg_pos={0 0}, upload=477953]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:30:58.683913139 (545h1m58.554996174s) [upload=477953]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:30:58.683920063 (545h1m58.555003108s) [upload=477953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:30:58.984226622 (545h1m58.855309658s) [upload=477953]: Done uploading segments... Dec 28 08:30:58.984252682 (545h1m58.855335727s) [upload=477953]: Upload finished. err: <nil> Duration: 30.003046494s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 16.883683755s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:31:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 16.883710926s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [......XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X..] duration: 4.168157375s trace: https://jaeger.dev.storj.io/trace/6fd820859e8b0174 FAIL (2025-12-28 08:30:39 UTC) orbiter warning: context deadline exceeded Duration: 4.168185459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X......XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X.] duration: 3.46665633s trace: https://jaeger.dev.storj.io/trace/6fd820859e8b0174 FAIL (2025-12-28 08:30:23 UTC) orbiter warning: context deadline exceeded Duration: 3.466685236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.X] duration: 5.000763815s trace: https://jaeger.dev.storj.io/trace/6fd820859e8b0174 FAIL (2025-12-28 08:30:08 UTC) orbiter warning: context deadline exceeded Duration: 5.000800665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.996351608s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:30:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.996381606s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X.] duration: 6.178376971s trace: https://jaeger.dev.storj.io/trace/15cda7d0bdc80f19 FAIL (2025-12-28 08:29:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:29:24.087270224 (545h0m23.95835326s) [upload=477933]: Starting upload Dec 28 08:29:24.087366838 (545h0m23.958449883s) [upload=477933]: Done starting upload Dec 28 08:29:24.087443153 (545h0m23.958526198s) [upload=477933]: Uploading segments... Dec 28 08:29:24.087687940 (545h0m23.958770995s) [upload=477933]: Got next segment. Inline: false Dec 28 08:29:24.087734158 (545h0m23.958817203s) [upload=477933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:29:24.087740419 (545h0m23.958823454s) [upload=477933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:29:24.336772493 (545h0m24.207855528s) [seg_pos={0 0}, upload=477933]: Begin upload segment... Dec 28 08:29:24.336813612 (545h0m24.207896657s) [seg_pos={0 0}, upload=477933]: Done begin upload segment. Dec 28 08:29:24.336840612 (545h0m24.207923657s) [noise=true , address=145.79.227.9:4365 , piece_id=DVS64ARTUJHSV6KJGNFANGQIIK4ZNYMWPKXPGLBKDMHHK7N7W27A, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.336883775 (545h0m24.20796681s) [noise=true , address=145.79.227.6:4258 , piece_id=JMJEPQHFSX3CLBX3ZP4JP56O2TNLNQH55UQN6IE3G2QJ45KVY54A, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.337331087 (545h0m24.208414132s) [noise=true , address=145.79.227.7:4293 , piece_id=I2V4CFKT4BVIA74O4VCSRFVYX7F2A2YTJPX2CJEMGN4BT5N6AMYA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.337718404 (545h0m24.208801439s) [noise=true , address=145.79.227.15:4876 , piece_id=JHQ4424JFGT2QFY3FTNTPXWZMBYTZOFTCMFB5DHMNFEEMS4YTNAA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.338169924 (545h0m24.209252959s) [noise=true , address=45.91.116.11:4120 , piece_id=D7WED25TNN2QD6424PEVV7AXEVRHMBNGVEMWXRJ3FY2M6I6GPZ3A, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.338542112 (545h0m24.209625157s) [noise=true , address=145.79.227.4:4185 , piece_id=NUN3TM6XLROHID5VILXQPBAKA4TE4T442IWGIHTCVPFZIXGKM5MA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.338957624 (545h0m24.210040659s) [noise=true , address=45.91.116.9:4261 , piece_id=IDCIBMCF3GFOW6X3ED7IGQWA7Y2Y5VXAEPH7OIUVQG7ZJJHE3ZJA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.339377623 (545h0m24.210460668s) [noise=true , address=145.79.227.19:4078 , piece_id=HI24U4RPRWU3MWTWHZRZ5D7IL23WVZOBRSCROI4EY2FTO33RW5MQ, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.340441419 (545h0m24.211524464s) [noise=true , address=145.79.227.13:4802 , piece_id=WU5QK3RLH4BBRSOOHLZXICPW2R3PTT6EKZOQOU2TJKTTDWOX4XXQ, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.340727073 (545h0m24.211810118s) [noise=true , address=45.91.116.5:4298 , piece_id=S5WFKMFF5OJJ64YFEQWCYFD7NLHVNJ2APVYGXOKKYILF2YR723MA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.341580489 (545h0m24.212663524s) [noise=true , address=45.91.116.10:4082 , piece_id=3MW4VQ5NIR3TG5GSJAKX7SJASHKETTWXFXWYVFVFBB4NMNARMHBA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.342743713 (545h0m24.213826748s) [noise=true , address=145.79.227.12:4767 , piece_id=LHXAT4AL2V6QIAK5PCLMZT766347NVOOEIC4USDQWEF4PQHMS5AA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.343912449 (545h0m24.214995484s) [noise=true , address=145.79.227.30:4762 , piece_id=M7OXN4EP22P4JMW244YQQHVDZOBNIP5RPGEBSIIGQO6KW3MKLVFA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.345011692 (545h0m24.216094737s) [noise=true , address=145.79.227.38:4977 , piece_id=WQGN3ZTFSAYJHWGVYTQ25AGR4QN675FDSZP4WKOJFZFPGBM655HA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.346560060 (545h0m24.217643106s) [noise=true , address=145.79.227.3:4151 , piece_id=ZW25WDPN4LQQ6S4YI5TBHNUR3NGGZJC357GSDMBQBB2V7DAFNIRQ, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.347996766 (545h0m24.219079811s) [noise=true , address=45.91.116.12:4227 , piece_id=6LROQ2XSXKMAP2PEI4QEQYR7SSGHMRI7BBPLKBS4FBQREBZN2FSQ, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:25.959033121 (545h0m25.830116156s) [upload=477933]: Next returned nil segment Dec 28 08:29:25.959052067 (545h0m25.830135102s) [upload=477933]: Waiting for error group managing segments... Dec 28 08:29:26.344967112 (545h0m26.216050157s) [noise=true , address=145.79.227.19:4078 , piece_id=HI24U4RPRWU3MWTWHZRZ5D7IL23WVZOBRSCROI4EY2FTO33RW5MQ, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.347135410 (545h0m26.218218445s) [noise=true , address=145.79.227.9:4365 , piece_id=DVS64ARTUJHSV6KJGNFANGQIIK4ZNYMWPKXPGLBKDMHHK7N7W27A, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.347782763 (545h0m26.218865798s) [noise=true , address=145.79.227.4:4185 , piece_id=NUN3TM6XLROHID5VILXQPBAKA4TE4T442IWGIHTCVPFZIXGKM5MA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.348984581 (545h0m26.220067616s) [noise=true , address=145.79.227.38:4977 , piece_id=WQGN3ZTFSAYJHWGVYTQ25AGR4QN675FDSZP4WKOJFZFPGBM655HA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.351255917 (545h0m26.222338952s) [noise=true , address=145.79.227.12:4767 , piece_id=LHXAT4AL2V6QIAK5PCLMZT766347NVOOEIC4USDQWEF4PQHMS5AA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.358426651 (545h0m26.229509686s) [noise=true , address=145.79.227.7:4293 , piece_id=I2V4CFKT4BVIA74O4VCSRFVYX7F2A2YTJPX2CJEMGN4BT5N6AMYA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.359227536 (545h0m26.230310581s) [noise=true , address=145.79.227.13:4802 , piece_id=WU5QK3RLH4BBRSOOHLZXICPW2R3PTT6EKZOQOU2TJKTTDWOX4XXQ, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.360921612 (545h0m26.232004648s) [noise=true , address=145.79.227.6:4258 , piece_id=JMJEPQHFSX3CLBX3ZP4JP56O2TNLNQH55UQN6IE3G2QJ45KVY54A, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.362313623 (545h0m26.233396658s) [noise=true , address=145.79.227.15:4876 , piece_id=JHQ4424JFGT2QFY3FTNTPXWZMBYTZOFTCMFB5DHMNFEEMS4YTNAA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.388831368 (545h0m26.259914413s) [noise=true , address=145.79.227.3:4151 , piece_id=ZW25WDPN4LQQ6S4YI5TBHNUR3NGGZJC357GSDMBQBB2V7DAFNIRQ, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.563828254 (545h0m26.434911299s) [noise=true , address=45.91.116.11:4120 , piece_id=D7WED25TNN2QD6424PEVV7AXEVRHMBNGVEMWXRJ3FY2M6I6GPZ3A, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.583462742 (545h0m26.454545777s) [noise=true , address=45.91.116.12:4227 , piece_id=6LROQ2XSXKMAP2PEI4QEQYR7SSGHMRI7BBPLKBS4FBQREBZN2FSQ, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.620093454 (545h0m26.491176489s) [noise=true , address=45.91.116.10:4082 , piece_id=3MW4VQ5NIR3TG5GSJAKX7SJASHKETTWXFXWYVFVFBB4NMNARMHBA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.620121107 (545h0m26.491204152s) [seg_pos={0 0}, upload=477933]: Segment reached optimal threshold of 13 pieces. Dec 28 08:29:26.620393886 (545h0m26.491476931s) [noise=true , address=45.91.116.5:4298 , piece_id=S5WFKMFF5OJJ64YFEQWCYFD7NLHVNJ2APVYGXOKKYILF2YR723MA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:29:26.620404947 (545h0m26.491487992s) [noise=true , address=45.91.116.5:4298 , piece_id=S5WFKMFF5OJJ64YFEQWCYFD7NLHVNJ2APVYGXOKKYILF2YR723MA, seg_pos={0 0}, upload=477933]: Long tail cancellation due to optimal threshold Dec 28 08:29:26.620578057 (545h0m26.491661092s) [noise=true , address=45.91.116.9:4261 , piece_id=IDCIBMCF3GFOW6X3ED7IGQWA7Y2Y5VXAEPH7OIUVQG7ZJJHE3ZJA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 08:29:26.620587134 (545h0m26.491670169s) [noise=true , address=45.91.116.9:4261 , piece_id=IDCIBMCF3GFOW6X3ED7IGQWA7Y2Y5VXAEPH7OIUVQG7ZJJHE3ZJA, seg_pos={0 0}, upload=477933]: Long tail cancellation due to optimal threshold Dec 28 08:29:26.620615078 (545h0m26.491698113s) [noise=true , address=145.79.227.30:4762 , piece_id=M7OXN4EP22P4JMW244YQQHVDZOBNIP5RPGEBSIIGQO6KW3MKLVFA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 08:29:26.620622511 (545h0m26.491705547s) [noise=true , address=145.79.227.30:4762 , piece_id=M7OXN4EP22P4JMW244YQQHVDZOBNIP5RPGEBSIIGQO6KW3MKLVFA, seg_pos={0 0}, upload=477933]: Long tail cancellation due to optimal threshold Dec 28 08:29:26.620635856 (545h0m26.491718902s) [seg_pos={0 0}, upload=477933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:29:26.620721941 (545h0m26.491804986s) [upload=477933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:29:26.620727131 (545h0m26.491810166s) [upload=477933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:29:26.916931198 (545h0m26.788014243s) [upload=477933]: Done uploading segments... Dec 28 08:29:26.916957439 (545h0m26.788040484s) [upload=477933]: Upload finished. err: <nil> Duration: 6.1783981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...X] duration: 30.003867022s trace: https://jaeger.dev.storj.io/trace/15cda7d0bdc80f19 FAIL (2025-12-28 08:29:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:29:24.087270224 (545h0m23.95835326s) [upload=477933]: Starting upload Dec 28 08:29:24.087366838 (545h0m23.958449883s) [upload=477933]: Done starting upload Dec 28 08:29:24.087443153 (545h0m23.958526198s) [upload=477933]: Uploading segments... Dec 28 08:29:24.087687940 (545h0m23.958770995s) [upload=477933]: Got next segment. Inline: false Dec 28 08:29:24.087734158 (545h0m23.958817203s) [upload=477933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:29:24.087740419 (545h0m23.958823454s) [upload=477933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:29:24.336772493 (545h0m24.207855528s) [seg_pos={0 0}, upload=477933]: Begin upload segment... Dec 28 08:29:24.336813612 (545h0m24.207896657s) [seg_pos={0 0}, upload=477933]: Done begin upload segment. Dec 28 08:29:24.336840612 (545h0m24.207923657s) [noise=true , address=145.79.227.9:4365 , piece_id=DVS64ARTUJHSV6KJGNFANGQIIK4ZNYMWPKXPGLBKDMHHK7N7W27A, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.336883775 (545h0m24.20796681s) [noise=true , address=145.79.227.6:4258 , piece_id=JMJEPQHFSX3CLBX3ZP4JP56O2TNLNQH55UQN6IE3G2QJ45KVY54A, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.337331087 (545h0m24.208414132s) [noise=true , address=145.79.227.7:4293 , piece_id=I2V4CFKT4BVIA74O4VCSRFVYX7F2A2YTJPX2CJEMGN4BT5N6AMYA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.337718404 (545h0m24.208801439s) [noise=true , address=145.79.227.15:4876 , piece_id=JHQ4424JFGT2QFY3FTNTPXWZMBYTZOFTCMFB5DHMNFEEMS4YTNAA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.338169924 (545h0m24.209252959s) [noise=true , address=45.91.116.11:4120 , piece_id=D7WED25TNN2QD6424PEVV7AXEVRHMBNGVEMWXRJ3FY2M6I6GPZ3A, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.338542112 (545h0m24.209625157s) [noise=true , address=145.79.227.4:4185 , piece_id=NUN3TM6XLROHID5VILXQPBAKA4TE4T442IWGIHTCVPFZIXGKM5MA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.338957624 (545h0m24.210040659s) [noise=true , address=45.91.116.9:4261 , piece_id=IDCIBMCF3GFOW6X3ED7IGQWA7Y2Y5VXAEPH7OIUVQG7ZJJHE3ZJA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.339377623 (545h0m24.210460668s) [noise=true , address=145.79.227.19:4078 , piece_id=HI24U4RPRWU3MWTWHZRZ5D7IL23WVZOBRSCROI4EY2FTO33RW5MQ, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.340441419 (545h0m24.211524464s) [noise=true , address=145.79.227.13:4802 , piece_id=WU5QK3RLH4BBRSOOHLZXICPW2R3PTT6EKZOQOU2TJKTTDWOX4XXQ, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.340727073 (545h0m24.211810118s) [noise=true , address=45.91.116.5:4298 , piece_id=S5WFKMFF5OJJ64YFEQWCYFD7NLHVNJ2APVYGXOKKYILF2YR723MA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.341580489 (545h0m24.212663524s) [noise=true , address=45.91.116.10:4082 , piece_id=3MW4VQ5NIR3TG5GSJAKX7SJASHKETTWXFXWYVFVFBB4NMNARMHBA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.342743713 (545h0m24.213826748s) [noise=true , address=145.79.227.12:4767 , piece_id=LHXAT4AL2V6QIAK5PCLMZT766347NVOOEIC4USDQWEF4PQHMS5AA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.343912449 (545h0m24.214995484s) [noise=true , address=145.79.227.30:4762 , piece_id=M7OXN4EP22P4JMW244YQQHVDZOBNIP5RPGEBSIIGQO6KW3MKLVFA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.345011692 (545h0m24.216094737s) [noise=true , address=145.79.227.38:4977 , piece_id=WQGN3ZTFSAYJHWGVYTQ25AGR4QN675FDSZP4WKOJFZFPGBM655HA, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.346560060 (545h0m24.217643106s) [noise=true , address=145.79.227.3:4151 , piece_id=ZW25WDPN4LQQ6S4YI5TBHNUR3NGGZJC357GSDMBQBB2V7DAFNIRQ, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:24.347996766 (545h0m24.219079811s) [noise=true , address=45.91.116.12:4227 , piece_id=6LROQ2XSXKMAP2PEI4QEQYR7SSGHMRI7BBPLKBS4FBQREBZN2FSQ, seg_pos={0 0}, upload=477933]: Uploading piece... Dec 28 08:29:25.959033121 (545h0m25.830116156s) [upload=477933]: Next returned nil segment Dec 28 08:29:25.959052067 (545h0m25.830135102s) [upload=477933]: Waiting for error group managing segments... Dec 28 08:29:26.344967112 (545h0m26.216050157s) [noise=true , address=145.79.227.19:4078 , piece_id=HI24U4RPRWU3MWTWHZRZ5D7IL23WVZOBRSCROI4EY2FTO33RW5MQ, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.347135410 (545h0m26.218218445s) [noise=true , address=145.79.227.9:4365 , piece_id=DVS64ARTUJHSV6KJGNFANGQIIK4ZNYMWPKXPGLBKDMHHK7N7W27A, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.347782763 (545h0m26.218865798s) [noise=true , address=145.79.227.4:4185 , piece_id=NUN3TM6XLROHID5VILXQPBAKA4TE4T442IWGIHTCVPFZIXGKM5MA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.348984581 (545h0m26.220067616s) [noise=true , address=145.79.227.38:4977 , piece_id=WQGN3ZTFSAYJHWGVYTQ25AGR4QN675FDSZP4WKOJFZFPGBM655HA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.351255917 (545h0m26.222338952s) [noise=true , address=145.79.227.12:4767 , piece_id=LHXAT4AL2V6QIAK5PCLMZT766347NVOOEIC4USDQWEF4PQHMS5AA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.358426651 (545h0m26.229509686s) [noise=true , address=145.79.227.7:4293 , piece_id=I2V4CFKT4BVIA74O4VCSRFVYX7F2A2YTJPX2CJEMGN4BT5N6AMYA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.359227536 (545h0m26.230310581s) [noise=true , address=145.79.227.13:4802 , piece_id=WU5QK3RLH4BBRSOOHLZXICPW2R3PTT6EKZOQOU2TJKTTDWOX4XXQ, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.360921612 (545h0m26.232004648s) [noise=true , address=145.79.227.6:4258 , piece_id=JMJEPQHFSX3CLBX3ZP4JP56O2TNLNQH55UQN6IE3G2QJ45KVY54A, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.362313623 (545h0m26.233396658s) [noise=true , address=145.79.227.15:4876 , piece_id=JHQ4424JFGT2QFY3FTNTPXWZMBYTZOFTCMFB5DHMNFEEMS4YTNAA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.388831368 (545h0m26.259914413s) [noise=true , address=145.79.227.3:4151 , piece_id=ZW25WDPN4LQQ6S4YI5TBHNUR3NGGZJC357GSDMBQBB2V7DAFNIRQ, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.563828254 (545h0m26.434911299s) [noise=true , address=45.91.116.11:4120 , piece_id=D7WED25TNN2QD6424PEVV7AXEVRHMBNGVEMWXRJ3FY2M6I6GPZ3A, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.583462742 (545h0m26.454545777s) [noise=true , address=45.91.116.12:4227 , piece_id=6LROQ2XSXKMAP2PEI4QEQYR7SSGHMRI7BBPLKBS4FBQREBZN2FSQ, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.620093454 (545h0m26.491176489s) [noise=true , address=45.91.116.10:4082 , piece_id=3MW4VQ5NIR3TG5GSJAKX7SJASHKETTWXFXWYVFVFBB4NMNARMHBA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: <nil> Dec 28 08:29:26.620121107 (545h0m26.491204152s) [seg_pos={0 0}, upload=477933]: Segment reached optimal threshold of 13 pieces. Dec 28 08:29:26.620393886 (545h0m26.491476931s) [noise=true , address=45.91.116.5:4298 , piece_id=S5WFKMFF5OJJ64YFEQWCYFD7NLHVNJ2APVYGXOKKYILF2YR723MA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:29:26.620404947 (545h0m26.491487992s) [noise=true , address=45.91.116.5:4298 , piece_id=S5WFKMFF5OJJ64YFEQWCYFD7NLHVNJ2APVYGXOKKYILF2YR723MA, seg_pos={0 0}, upload=477933]: Long tail cancellation due to optimal threshold Dec 28 08:29:26.620578057 (545h0m26.491661092s) [noise=true , address=45.91.116.9:4261 , piece_id=IDCIBMCF3GFOW6X3ED7IGQWA7Y2Y5VXAEPH7OIUVQG7ZJJHE3ZJA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 08:29:26.620587134 (545h0m26.491670169s) [noise=true , address=45.91.116.9:4261 , piece_id=IDCIBMCF3GFOW6X3ED7IGQWA7Y2Y5VXAEPH7OIUVQG7ZJJHE3ZJA, seg_pos={0 0}, upload=477933]: Long tail cancellation due to optimal threshold Dec 28 08:29:26.620615078 (545h0m26.491698113s) [noise=true , address=145.79.227.30:4762 , piece_id=M7OXN4EP22P4JMW244YQQHVDZOBNIP5RPGEBSIIGQO6KW3MKLVFA, seg_pos={0 0}, upload=477933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 08:29:26.620622511 (545h0m26.491705547s) [noise=true , address=145.79.227.30:4762 , piece_id=M7OXN4EP22P4JMW244YQQHVDZOBNIP5RPGEBSIIGQO6KW3MKLVFA, seg_pos={0 0}, upload=477933]: Long tail cancellation due to optimal threshold Dec 28 08:29:26.620635856 (545h0m26.491718902s) [seg_pos={0 0}, upload=477933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:29:26.620721941 (545h0m26.491804986s) [upload=477933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:29:26.620727131 (545h0m26.491810166s) [upload=477933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:29:26.916931198 (545h0m26.788014243s) [upload=477933]: Done uploading segments... Dec 28 08:29:26.916957439 (545h0m26.788040484s) [upload=477933]: Upload finished. err: <nil> Duration: 30.003926626s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 11.81089525s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:29:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.810913735s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.XX......XX........X.XX..XXX.X.XX.XX.......X.XX.XX.......XX.] duration: 4.985914157s trace: https://jaeger.dev.storj.io/trace/7ca3ab3ea093590c FAIL (2025-12-28 08:29:02 UTC) orbiter warning: context deadline exceeded Duration: 4.98594186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.282020233s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:28:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.282046984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...XX....X.XX..X...XX.XX.XX.XX.XXX..X.XX.X.XX..X..XX....X...] duration: 5.729233149s trace: https://jaeger.dev.storj.io/trace/271dd2887f4dc157 FAIL (2025-12-28 08:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 6TCINXXUHHHWV43SP2NEHFREVRQY3WL2UOZN7G4WOVSJWHLGWFQA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: J5JCCUHL2ONGNPRR6KEXA3HBLSJYIUYCGGXO344SH5QHU2SU37IA) context deadline exceeded ======LOGS====== Dec 28 08:26:46.523470594 (544h57m46.394553639s) [upload=477890]: Starting upload Dec 28 08:26:46.523528214 (544h57m46.394611259s) [upload=477890]: Done starting upload Dec 28 08:26:46.523751990 (544h57m46.394835025s) [upload=477890]: Uploading segments... Dec 28 08:26:46.524857114 (544h57m46.395940159s) [upload=477890]: Got next segment. Inline: false Dec 28 08:26:46.524893363 (544h57m46.395976398s) [upload=477890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:26:46.524897892 (544h57m46.395980927s) [upload=477890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:26:46.767263743 (544h57m46.638346778s) [seg_pos={0 0}, upload=477890]: Begin upload segment... Dec 28 08:26:46.767317034 (544h57m46.638400069s) [seg_pos={0 0}, upload=477890]: Done begin upload segment. Dec 28 08:26:46.767351750 (544h57m46.638434785s) [noise=true , address=45.91.116.10:4084 , piece_id=QEDSKUWCP72LJHZ2NUHN2R5DJJHK4VASX3DN5DOWZV3VLTBNVH5Q, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.767411364 (544h57m46.638494409s) [noise=true , address=145.79.227.52:4446 , piece_id=5HCN2SG7JQD5DQQLIHHEJLKWHLXKLWX5J45XWMP457YZNIPREBZA, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.767966771 (544h57m46.639049816s) [noise=true , address=77.90.31.21:4877 , piece_id=J5JCCUHL2ONGNPRR6KEXA3HBLSJYIUYCGGXO344SH5QHU2SU37IA, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.768497512 (544h57m46.639580557s) [noise=true , address=145.79.227.6:4259 , piece_id=C22L4IR6UVXS26KV2CRMTTJD4BCI5RYIAHPNDJFDITOYBHCUHFGA, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.768687072 (544h57m46.639770107s) [noise=true , address=145.79.227.14:4840 , piece_id=IAO4L2DFGMDBZAIGOKTRTIIFQRVL2SGQVI4TX2LHWVFBT2OLEQ7A, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.769109057 (544h57m46.640192092s) [noise=true , address=145.79.227.38:4979 , piece_id=74WSKQXVNFHCO6YHUMJLU7JUPJ4HKDQI22A4FC4RUZRODLYRD7UA, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.769485803 (544h57m46.640568848s) [noise=true , address=77.90.31.14:4661 , piece_id=YC5HLS53OGLS6MIRDQ4IKGCO7NPF3BBZJTJDT77GMOHWU5T2YCPQ, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.769841121 (544h57m46.640924156s) [noise=true , address=45.91.116.13:4154 , piece_id=6TCINXXUHHHWV43SP2NEHFREVRQY3WL2UOZN7G4WOVSJWHLGWFQA, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.770197168 (544h57m46.641280203s) [noise=true , address=45.91.116.5:4298 , piece_id=U25GVEAKNH7MZ64WMPRRWXT3DX3QKCLNLODEFFKV2ISYBVASLRKQ, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.770394483 (544h57m46.641477528s) [noise=true , address=45.91.116.6:4973 , piece_id=IMDZ7HG4XV26Y3KTPDFOIHKIN4XF2S2DTWHI3YZTAYCSYSGHSM5A, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.770569587 (544h57m46.641652622s) [noise=true , address=77.90.31.12:4590 , piece_id=FU6FVVNZAPLRDDHPSSHHCD4IGWEBWVD7CSHOPJJ5O4TFHOKFWGUA, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.770863277 (544h57m46.641946322s) [noise=true , address=45.91.116.4:4940 , piece_id=5CVOTMGPL354A2MUHKCD7VUNYKI2YVLPYKOHZPASTEAU7SEITD6A, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.770912249 (544h57m46.641995284s) [noise=true , address=145.79.227.25:4619 , piece_id=QII7EN2VBATXO22T5ELDGVOTBJ4PQV5EILGW37IVFTRVDWTLZX4Q, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.771335125 (544h57m46.64241816s) [noise=true , address=77.90.31.10:4518 , piece_id=A43M42NHNIHFACF25WLUN6RJB4OWOUFY2R4NQBVTZZWBD27LQCRA, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.771475011 (544h57m46.642558046s) [noise=true , address=145.79.227.44:4194 , piece_id=6UGVEFWU4WNDVBXQFYFTZULOJERKHY6VQBDJU3TH2YZAGQRUNSKQ, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:46.771787536 (544h57m46.642870581s) [noise=true , address=145.79.227.17:4970 , piece_id=NYA5SUNDGXZFSFOX3WIRYWTDSR7WSKVLQPILC43H554PXJ3Y6IGA, seg_pos={0 0}, upload=477890]: Uploading piece... Dec 28 08:26:48.477078860 (544h57m48.348161896s) [upload=477890]: Next returned nil segment Dec 28 08:26:48.477099230 (544h57m48.348182265s) [upload=477890]: Waiting for error group managing segments... Dec 28 08:26:48.775407585 (544h57m48.64649062s) [noise=true , address=145.79.227.52:4446 , piece_id=5HCN2SG7JQD5DQQLIHHEJLKWHLXKLWX5J45XWMP457YZNIPREBZA, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:48.777222130 (544h57m48.648305175s) [noise=true , address=145.79.227.25:4619 , piece_id=QII7EN2VBATXO22T5ELDGVOTBJ4PQV5EILGW37IVFTRVDWTLZX4Q, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:48.777733975 (544h57m48.64881701s) [noise=true , address=145.79.227.6:4259 , piece_id=C22L4IR6UVXS26KV2CRMTTJD4BCI5RYIAHPNDJFDITOYBHCUHFGA, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:48.780391745 (544h57m48.65147478s) [noise=true , address=145.79.227.14:4840 , piece_id=IAO4L2DFGMDBZAIGOKTRTIIFQRVL2SGQVI4TX2LHWVFBT2OLEQ7A, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:48.784733303 (544h57m48.655816338s) [noise=true , address=145.79.227.38:4979 , piece_id=74WSKQXVNFHCO6YHUMJLU7JUPJ4HKDQI22A4FC4RUZRODLYRD7UA, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:48.802495936 (544h57m48.673578981s) [noise=true , address=145.79.227.17:4970 , piece_id=NYA5SUNDGXZFSFOX3WIRYWTDSR7WSKVLQPILC43H554PXJ3Y6IGA, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:48.969203489 (544h57m48.840286524s) [noise=true , address=77.90.31.14:4661 , piece_id=YC5HLS53OGLS6MIRDQ4IKGCO7NPF3BBZJTJDT77GMOHWU5T2YCPQ, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:49.008601239 (544h57m48.879684284s) [noise=true , address=45.91.116.6:4973 , piece_id=IMDZ7HG4XV26Y3KTPDFOIHKIN4XF2S2DTWHI3YZTAYCSYSGHSM5A, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:49.050711255 (544h57m48.92179429s) [noise=true , address=77.90.31.12:4590 , piece_id=FU6FVVNZAPLRDDHPSSHHCD4IGWEBWVD7CSHOPJJ5O4TFHOKFWGUA, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:49.083657341 (544h57m48.954740376s) [noise=true , address=77.90.31.21:4877 , piece_id=J5JCCUHL2ONGNPRR6KEXA3HBLSJYIUYCGGXO344SH5QHU2SU37IA, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:49.112191495 (544h57m48.98327454s) [noise=true , address=45.91.116.10:4084 , piece_id=QEDSKUWCP72LJHZ2NUHN2R5DJJHK4VASX3DN5DOWZV3VLTBNVH5Q, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:49.734518833 (544h57m49.605601868s) [noise=true , address=45.91.116.13:4154 , piece_id=6TCINXXUHHHWV43SP2NEHFREVRQY3WL2UOZN7G4WOVSJWHLGWFQA, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:49.808472875 (544h57m49.67955592s) [noise=true , address=45.91.116.4:4940 , piece_id=5CVOTMGPL354A2MUHKCD7VUNYKI2YVLPYKOHZPASTEAU7SEITD6A, seg_pos={0 0}, upload=477890]: Done uploading piece. err: <nil> Dec 28 08:26:49.808498514 (544h57m49.679581559s) [seg_pos={0 0}, upload=477890]: Segment reached optimal threshold of 13 pieces. Dec 28 08:26:49.808752768 (544h57m49.679835813s) [noise=true , address=77.90.31.10:4518 , piece_id=A43M42NHNIHFACF25WLUN6RJB4OWOUFY2R4NQBVTZZWBD27LQCRA, seg_pos={0 0}, upload=477890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 08:26:49.808759220 (544h57m49.679842255s) [noise=true , address=77.90.31.10:4518 , piece_id=A43M42NHNIHFACF25WLUN6RJB4OWOUFY2R4NQBVTZZWBD27LQCRA, seg_pos={0 0}, upload=477890]: Long tail cancellation due to optimal threshold Dec 28 08:26:49.808960193 (544h57m49.680043228s) [noise=true , address=45.91.116.5:4298 , piece_id=U25GVEAKNH7MZ64WMPRRWXT3DX3QKCLNLODEFFKV2ISYBVASLRKQ, seg_pos={0 0}, upload=477890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:26:49.808967106 (544h57m49.680050141s) [noise=true , address=45.91.116.5:4298 , piece_id=U25GVEAKNH7MZ64WMPRRWXT3DX3QKCLNLODEFFKV2ISYBVASLRKQ, seg_pos={0 0}, upload=477890]: Long tail cancellation due to optimal threshold Dec 28 08:26:49.809068890 (544h57m49.680151925s) [noise=true , address=145.79.227.44:4194 , piece_id=6UGVEFWU4WNDVBXQFYFTZULOJERKHY6VQBDJU3TH2YZAGQRUNSKQ, seg_pos={0 0}, upload=477890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 08:26:49.809074290 (544h57m49.680157325s) [noise=true , address=145.79.227.44:4194 , piece_id=6UGVEFWU4WNDVBXQFYFTZULOJERKHY6VQBDJU3TH2YZAGQRUNSKQ, seg_pos={0 0}, upload=477890]: Long tail cancellation due to optimal threshold Dec 28 08:26:49.809089880 (544h57m49.680172915s) [seg_pos={0 0}, upload=477890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:26:49.809172186 (544h57m49.680255221s) [upload=477890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:26:49.809178789 (544h57m49.680261824s) [upload=477890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:26:50.102431407 (544h57m49.973514442s) [upload=477890]: Done uploading segments... Dec 28 08:26:50.102454632 (544h57m49.973537677s) [upload=477890]: Upload finished. err: <nil> Duration: 5.729268877s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)